Improving documentation state

[Jean-Baptiste Boric]

Hi,

As mentioned on this mailing list, the current state of documentation isn't exactly as good as it ought to be (in particular, the wiki is getting quite stale in some parts). I gave it some thoughts and here are my ideas :

Could Doxygen be used to improve the current situation in specific cases? Things like Minix header libraries (fsdriver.h, chardriver.h, ...) and the IPC messages (ipc.h) seem to be good candidates for this. Documentation and code would be kept in sync naturally, at the cost of more verbose headers.

Minix manpages haven't seen much activity for some time, besides removing some of them due to the NetBSD userland transition. Besides possibly putting them back into style for the same things as above (not a fan of that idea, they run at a higher risk of staleness compared to Doxygen-like solutions), I didn't found a CGI interface to browse them on the web (there is http://www.minix3.org/manpages, but it seems to be a leftover from the Minix 3.1 days or before).

A nice bonus could be to make Doxygen (and/or man) output DokuWiki pages to import directly on the wiki. Having all the documentation compiled in one place would be nice, but generating wiki pages from source code kinda defeats the whole point of a wiki (a big banner saying that those wiki pages are generated from source code and thus shouldn't be directly modified would be required).

The reverse is also possible for man pages (putting them on the wiki and generating man pages from them once in a while for the Git repository), which would make them more accessible for editing by the community and much more wiki-like. Again, bonus point for regrouping documentation. Obviously, this should be limited to Minix-specific manpages, as the NetBSD ones are inherited from, well, NetBSD.

Any thoughts on those ideas? Others ideas are welcome too (this is about thinking outside the box, just saying improve the wiki doesn't count).

[David van Moolenbroek]

Hey,

General reply to your mail: while I completely agree that we should have this discussion, I think in the end right now all these proposals only serve to move the problem around. Much of the core developer documentation is not so much outdated as it is plain missing. Once it's written, it can be put in any form, but I think that's secondary. My intention is still to work on documenting the interfaces I've worked on (mainly the {block|char|fs|input|net}driver ones) as soon as I'm done with a first version of my PhD thesis. That should be, uh, soon.

In terms of adding to the discussion at hand: we are going to need some pages that not only describe interfaces, but also explain concepts. For example, one issue that keeps coming up is memory grants and safecopies. This needs to be documented, and doxygen is probably not the best way to do that - ideally there'd be some figures as part of the explanation. Another desperately needed page is one that describes the larger system architecture of MINIX3, preferably also with figures. On the other hand, I (now) think that spending time on documenting low-level IPC protocols is not as useful (as I once thought), especially if we can get to a situation where they are mostly wrapped by libraries.

Just for reference, we currently have the VFS readme file which is supposed to be synchronized from the source into the wiki. The fact that typing this reminds me that I still have to sync them since my last change, does suggest that this is currently not ideal, either. Also, I for one don't know, and never have known, how the manpages on the website were supposed to be kept in sync.

Regards,
David

[Jean-Baptiste Boric]

I think in the end right now all these proposals only serve to move the problem around.

I agree that none of my proposals is a one-stop solution. Point is, the wiki is not a one-stop solution either (I'd rather consult an API reference on Doxygen or in a man page than on a wiki), hence the "think outside the box" spirit.
 

For example, one issue that keeps coming up is memory grants and safecopies. This needs to be documented, and doxygen is probably not the best way to do that

 

What I'm proposing (for a well-defined subset of the Minix source code) is to have the wiki make high-level explanations and have Doxygen (or man pages or something else) document the public API in detail. Thus both of those tools are employed at what they're best at and none of them step on each other's toes.

The interface libraries you worked on seems to be good first candidates for that.

Another desperately needed page is one that describes the larger system architecture of MINIX3, preferably also with figures.

Any news on that fourth book edition Andrew? ;-)

On a more serious note, Tanenbaum's video recordings of his talks about Minix seem like a good starting point for writing that kind of documentation.

On the other hand, I (now) think that spending time on documenting low-level IPC protocols is not as useful (as I once thought), especially if we can get to a situation where they are mostly wrapped by libraries.

Until someone comes along and modify the IPC protocols and/or the libraries.

I did some time ago made a rather successful experiment to switch permissions checking from VFS to file servers. The VFS-FS protocol documentation on the wiki proved extremely useful (though again I think Doxygen would be more suited for the low-level details of that).

After all, one of the reasons NetBSD's huge collection of ports exists in the first place is because pretty much *everything* is thoroughly documented.

Granted, it's nowhere not as important as documenting the library wrappers. But if it's in the Minix source code, someone someday is going to study/work on it, and that guy is going to enjoy having documentation on that part of Minix instead of reverse-engineering thousands of lines of code. So I think it's still relevant to properly document them in the (very) long term.

Also, I for one don't know, and never have known, how the manpages on the website were supposed to be kept in sync.

There's plenty of other stuff on minix3.org that are leftovers from the old days. Stuff that predates the new site (those manpages are only one of them) were just left in place. It's still possible to fall down the rabbit hole if one knows where to look (and have a glimpse of the Minix 2 days).

I'm not sure what to do with those (probably best to move them inside a olddocs/ directory). But having CGI man pages for all Minix 3 versions (or even all versions) is both desirable in my opinion and a low-hanging fruit.

[Jean-Baptiste Boric]

Another desperately needed page is one that describes the larger system architecture of MINIX3, preferably also with figures.

I started to write pages about this, they are accessible on the bottom of the wiki's developers guide. Hopefully not too rough for introduction drafts.

But the more I think about it, the more I find the current layout of the table of contents to be simply inadequate. For example, kernel system calls, driver API, servers API and abstraction libraries are all blended together without much structure in the API documentation section ; the programming section is a collection of pages that have little in common each other...

I propose to create a new table of contents from scratch designed to accommodate all wiki developer documentation. A explicit goal is not to recreate OS Design and Implementation, so I think having sections by kernel, drivers, servers and user-land reflects nicely the architecture of Minix 3. Besides, as a developers guide it's supposed to be reference documentation with enough hand-holding for newcomers.

Here's a rough idea of what I have in mind :

1. Introduction
 a. Overview of Minix 3 architecture
 b. Tracking Minix development with Git
 c. Building/cross-compiling Minix
 d. Tests
 e. Contributing back
 ...
2. Machine-dependent
 a. i386
 b. ARM
3. Micro-kernel
 a. Overview of Minix micro-kernel
 b. Kernel system calls
 c. Message passing
 d. Memory grants
 ...
4. Drivers
 a. Overview of Minix drivers
 b. Tutorial for writing a driver
 c. Block devices
 d. Generic character devices
 e. Network cards
 f. I2C bus
 g. Real time clock (RTC)
 ...
5. Servers
 a. Overview of Minix servers
 b. Core servers
  i. Data store
  ii. Reincarnation server
  iii. Process manager
  iv. Scheduler
  v. Virtual memory
  ...
 c. File systems
  i. VFS server
  ii. File servers
   1. Minix file server (mfs)
   2. Ext2 file server (ext2)
   3. ISO 9660 file server (isofs)
   4. Pipe file server
  iii. VTreeFS library
  iv. File system specifications
   1. Minix file system
   ...
6. User-land
 a. Overview of Minix userland
 b. NetBSD userland
 c. Minix userland
 d. pkgsrc
...

Any thoughts on this?

[David van Moolenbroek]

Hey man,

Could Doxygen be used to improve the current situation in specific cases?

We (Lionel, stux, myself) discussed this on IRC a bit, and doxygen has a couple of problems: it is yet another piece of infrastructure, its implementation is GPL (so it can't be included in the base system), and -worst of all- it would only cause divergence with NetBSD. We want to avoid changing NetBSD headers or source files whenever possible, so we cannot include doxygen-style documentation in their stuff, and effectively that means that some functions would be documented with manual pages, and some functions would be documented through doxygen. That would just be inconvenient. Our preliminary conclusion is that it would be best to stick with manual pages, and indeed document anything suitable in manual pages rather than on the wiki. Lionel is working on infrastructure to convert manual pages into html, so that we can also access documentation from a website; as an added bonus, this will supersede the stale 3.1.0 manual pages.

Here's a rough idea of what I have in mind :

[..]

Any thoughts on this?

Go for it! :)

Regards,
David

[Sambuc Lionel]

Hi all,


I have a added a new subdomain as well as adapted a script to convert man pages to html / ascii / pdf from the FreeBSD web site.

This is accessible at: http://man.minix3.org.

At this time, I am not yet able to provide a to-the-commit up-to-date “current” version. This will require some more work, so I have made the choice to limit the currently available set of man pages to official releases.

If some can come up with an idea on how to get the equivalent of the /usr/man directory from the source tree, without requiring a full build, I can add that as a commit-hook on the server, and then provide a “current” revision from it.

Jean-Baptiste, awesome job on the wiki, I love it.


Regards,

Lionel

> --
> You received this message because you are subscribed to the Google Groups "minix3" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to minix3+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jean-Baptiste Boric]

If some can come up with an idea on how to get the equivalent of the /usr/man directory from the source tree, without requiring a full build, I can add that as a commit-hook on the server, and then provide a “current” revision from it.

How about find . -name "*.[0-9]" | grep -v "./tests/" , and possibly some shuffling around to put them in the correct directories?