Aggregating module documentation

[marbux]

Hi, All,

A useful component for the web site would be an aggregation of the
documentation for all batteries with a table of contents linking to
each module's documentation. Better yet if the documentation component
of the site were available as a separate download and as a separately
installable dist. The links of accompanying text might also indicate
the platforms for which dists are available.

Such an approach would be helpful to users who are trying to decide
whether to download and install a given module's dist as well as for
those who want to look up something in the documentation by providing
a single page target for navigation to the desired module's
documentation.

This is a subproject I'd be interested in working on.

DIFFICULTIES

-- Some module documentation (e.g., that for IUP) is framed HTML that
creates search barriers.

-- Quality and formatting of documentation is variable; e.g., in some
cases markdown would need to be converted to HTML.

-- Possible lack of a standard for location of module documentation in
LuaDist installations? (I haven't checked.)

-- Other difficulties or ideas for a better approach?

Best regards,

Paul

[Peter Drahoš]

On Sat, Mar 23, 2013 at 10:01 PM, marbux <mar...@gmail.com> wrote:

Hi, All,

A useful component for the web site would be an aggregation of the
documentation for all batteries with a table of contents linking to
each module's documentation. Better yet if the documentation component
of the site were available as a separate download and as a separately
installable dist. The links of accompanying text might also indicate
the platforms for which dists are available.

This can be done using the manifest we discussed earlier. LuaDist will mark all files it installs into dist.info file. There is a relevant section for documentation which is usually installed into \share\[module]\doc directory. There really is no need to separate the files into separate modules as LuaDist can optionally skip installing these "components" of the module.

For example:

Install only Lua runtime, nothing else

$ luadist _test install lua -components=Runtime

Install headers, docs and import libraries

$ luadist _test install lua -components=Runtime,Documentation,Header,Library 

Such an approach would be helpful to users who are trying to decide
whether to download and install a given module's dist as well as for
those who want to look up something in the documentation by providing
a single page target for navigation to the desired module's
documentation.

Providing direct access to the documentation is excellent idea. If we combine it with some unified style for each module then this can be a very powerful resource for users to look for information. 

This is a subproject I'd be interested in working on.

DIFFICULTIES

-- Some module documentation (e.g., that for IUP) is framed HTML that
creates search barriers.

 

We are taking role of the maintainers .. so we do not need to rely on the authors to conform to our standards but we need to do the work ourselves. If the work is the pulled into IUP .. even better.

 

-- Quality and formatting of documentation is variable; e.g., in some
cases markdown would need to be converted to HTML.

I still think we would benefit greatly if we keep documentation in Lua accessible form and generate HTML, Markdown and other documents from that. For example the website could actually use LuaDist to access the documentation of all installed modules and generate the HTML on the fly .. it would be basically the same on the desktop. A minimal server can be written that would serve all the documentation. A lot of work but I can dream ...

 

-- Possible lack of a standard for location of module documentation in
LuaDist installations? (I haven't checked.)

This is more or less standardized when using the macros LuaDist provides. However if we would follow on the above idea then documentation and even tests could be accessed using the require functionality in lua. Then the placement would be determined by package.path as for any other module.

Eg. to get documentation for socket we could do something like:

local ldoc = require "ldoc" --ldoc extended to support generation from "standard documentation format"

local ls = require "doc.luasocket" --returns luasocket documentation as a structured table

print( ldoc.doc2html(ls) ) -- generate html form

-- Other difficulties or ideas for a better approach?

The only limiting factor is that the standards simply do not exist. We have the tools, just not the standards that would apply them. It would still require quite a lot of effort to convert the existing modules. For a small selection of core modules this would be possible .. hopefully module developers would catch on and adopt.

 

NOTE: LuaRocks does not handle docs as far as I know but it could install the require-based docs without any modification. Same applies to linux distros .. the require mechanism is universal, lets use it.

pd

[marbux]

On Sat, Mar 23, 2013 at 2:24 PM, Peter Drahoš <dra...@gmail.com> wrote:

> The only limiting factor is that the standards simply do not exist. We have
> the tools, just not the standards that would apply them. It would still
> require quite a lot of effort to convert the existing modules. For a small
> selection of core modules this would be possible .. hopefully module
> developers would catch on and adopt.
>
> NOTE: LuaRocks does not handle docs as far as I know but it could install
> the require-based docs without any modification. Same applies to linux
> distros .. the require mechanism is universal, lets use it.

Peter, what you have written makes a lot of sense to me, although
multiple output formats would seem as a practical matter to require
working with an existing converter, e.g., the multi-platform Pandoc.

Best regards,

Paul

[Andrew Starks]

On Sun, Mar 24, 2013 at 12:10 AM, marbux <mar...@gmail.com> wrote:

> Peter, what you have written makes a lot of sense to me, although
> multiple output formats would seem as a practical matter to require
> working with an existing converter, e.g., the multi-platform Pandoc.
>
> Best regards,
>
> Paul
>

LDoc does this, no? Multimarkdown does, as well.

[steve donovan]

On Sun, Mar 24, 2013 at 2:31 PM, Andrew Starks <andrew...@trms.com> wrote:
> LDoc does this, no? Multimarkdown does, as well.

Sure, to some extent - pandoc can turn anything to anything, but I
find myself relying on LDoc's abilities to prettify Lua code blocks
and reference other sections.

Existing documentation is often already in HTML, which is awkward to
'decompile' as Paul M points out.

Peter makes an interesting point about require() being the universal
fetcher. When I prepared LDoc for LuaRocks I made the style sheets
and templates into Lua modules that just returned strings, so they
could sit with the code on the Lua module path.

[David Manura]

On Sat, Mar 23, 2013 at 5:24 PM, Peter Drahoš <dra...@gmail.com> wrote:

On Sat, Mar 23, 2013 at 10:01 PM, marbux <mar...@gmail.com> wrote:

A useful component for the web site would be an aggregation of the
documentation for all batteries with a table of contents linking to
each module's documentation. Better yet if the documentation component
of the site were available as a separate download and as a separately
installable dist.

My previous thoughts on this were collected at http://lua-users.org/wiki/DocumentingLuaCode .

I think it's helpful to define the bare minimum requirements for LfW/LuaDist documentation, get that operational, and proceed incrementally from there.

(1) We need a manner to systematically extract documentation from a package.  If there's a README.txt/md in the root folder, it's safe to say you want at least that, and github itself makes that assumption.  Likewise for ldoc.    Help from the package metadata may also be warranted.

(2) We need to be able to systematically convert documentation for all provided packages into (at least) HTML, all linked from a single index page.  This is readily achievable with documentation provided in README.txt, markdown, and ldoc formats.  HTML works too, although HTML as an input format has a tendency to over-specify on presentation, leading to inconsistencies between packages (e.g. frames as mentioned), but ugly documentation is better than no documentation though, and ugly documentation can always be improved later.  Perhaps we can build on the ldoc tool to handle merging of "foreign" documentation formats into the output.

(3) We probably want documentation to be portable: re-rendering copies of the documentation at new locations while properly preserving links.  For example, the canonical URL for the lpeg.P function is [1], but that link isn't guaranteed to be stable between versions, we might render that documentation on the LfW website, the user might locally install the documentation on their machine, and we may want to render to formats other than HTML as well.  This is an issue if documentation for two modules have links between them since we probably want to be able to systematically rewrite those links to refer to the local documentation.  Perl and Python communities have handled this well, and I understand ldoc likewise offers something like this too (@{ref}), but we need to keep in mind that many modules don't use ldoc (e.g. mine never had) and perhaps never will without strong encouragement.  However, it would not be impossible to likewise support such linking with documentation provided in multiple formats (README.txt, markdown, ldoc, and HTML) if certain basic linking conventions are followed within each of these formats, conventions that LfW/LuaDist could encourage.

(4) Given a certain module name (e.g. as passed to the `require` function), I understand we want to be able to systematically infer its corresponding documentation, as Peter noted.  I'd go even further to suggest we'd want to infer the location of documentation on an individual function or method in a module (e.g. perhaps rendered as an HTML anchor link), although omission of this is not a show stopper.  With this basic mechanism in place, editors and IDE's can be updated to take advantage of it contextually, either statically or at runtime.  Fabien had been involved in work in this area too in terms of IntelliSense and type inference like things and may have better ideas.

(5) We should have some recommendations for best practices as to what to include in the documentation and its overall content structure (e.g. sections on installation, dependencies, reference, license, examples/synopsis, etc.)  Perl POD has a long history of recommending various sections to include, and some good suggestions can be taken from there.  The place to start is in the core LfW modules, and where these are lacking, we may need to work with those module authors.  I'd be willing to offer help in such documentation reviews.

(6) The documentation conversion should properly satisfy any licensing requirements, such as ensuring the copyright and/or license are included in the rendered documentation.

(7) Any others?

Much of the above can be done *TODAY* and doesn't rely on unrealistic assumptions like that we'll convert 100's of modules all over to ldoc (as I had once tried with POD and reST).  The results may not be initially pretty, but any defects can be worked on in an incremental manner (e.g. #5).  #4 and perhaps #3 may require the most imagination though and must done in a way that people don't object to as too intrusive.

In some of my recent modules (at least those that are small such as [2]), I've taken a very bare bones approach to documentation, but it's one I think is still effective and that LfW/LuaDist wouldn't object to.  I've documented these modules merely with a single README.txt file in the root folder of the package, where the format is either text or markdown, and the content structure roughly follows the style inspired by Perl module documentation (and not too unlike [1] either).  github recognizes it and renders it as HTML (#1 & #2) and makes it forkable (#3).  Unfortunately, linking and retrieval (#3 & #4) are a bit lacking apart from a more structured formats I prefer, but if we follow some conventions, it should still be achievable.  For example, I see no reason why if someone writes the code "local FS = require 'fileslurp'; FS.readfile(...)" that the text editor shouldn't be able to readily infer that that "readfile" is documented on line 24 of that README.txt, if certain documentation conventions are followed and understood.  I'm not saying this documentation style is best for all modules (it probably is not) but rather that it is one style among multiple, and a simple one at that, that should work well within the context of the requirements above.

[1] http://www.inf.puc-rio.br/~roberto/lpeg/#op-p

[2] https://github.com/davidm/lua-file-slurp