Kernel docs: muddying the waters a bit

[Jonathan Corbet]

So I fear you all are going to hate me for this...

Asciidoc is a credible solution to the formatted documentation problem,
but it's not the only such; I'd like to be sure that we pick the right
one. I worry that asciidoc seems to be aimed mostly at small documents,
and that the project itself seems a little lifeless - it's not a good
sign when your main page's link to the repository has been dead for a long
time. (Asciidoctor seems more active, with the Github folks behind it,
but that means bringing Ruby into the picture).

An alternative we haven't really looked at yet is ReStructuredText (or
"RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is
YA simple markup scheme, remarkably similar to Markdown or Asciidoc;
Sphinx is a fairly sophisticated documentation system that uses RST.

I spent a few hours reworking the asciidoc patches to do RST instead, then
built a few template files' worth of docs. The result can be seen at:

http://static.lwn.net/kerneldoc/

It's very much a POC (however you might want to define the term), there's
lots of glitches, I chose a theme pretty much at random, etc. But it
shows that it can be done.

Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain
requirements there. It produces integrated, multi-file HTML natively,
with a TOC, an index, cross-file cross references, and more. It can make
things like function indexes. It claims output in epub, docbook, and man
(I've not yet messed with those). The path to PDF is via latex; clearly
the docbook path could be used too.

I used my same docproc hack to extract the comments here, mostly because I
had it at hand. We could go with Jani's separate-file approach if we
wanted. There's also a tool out there (called "breathe") that's meant to
turn doxygen-style comments into RST; I haven't had a chance to mess with
it. We *could* also write an extension to pull the comments directly in
Sphinx if there were a compelling reason to do so.

If anybody's curious, the work done to get this far is in:

git://git.lwn.net/linux.git doc/sphinx

but it looks suspiciously like the previous asciidoc patches, and, in any
case, it would have to be thrown out, publicly disowned, and replaced
before going any further with this, should that be what we decide to do.

So can we discuss? I'm not saying we have to use Sphinx, but, should we
choose not to, we should do so with open eyes and good reasons for the
course we do take. What do you all think?

jon

[Keith Packard]

Jonathan Corbet <cor...@lwn.net> writes:

> Asciidoc is a credible solution to the formatted documentation problem,
> but it's not the only such; I'd like to be sure that we pick the right
> one. I worry that asciidoc seems to be aimed mostly at small documents,
> and that the project itself seems a little lifeless - it's not a good
> sign when your main page's link to the repository has been dead for a long
> time. (Asciidoctor seems more active, with the Github folks behind it,
> but that means bringing Ruby into the picture).

I was surprised when one of the asciidoctor developers said that
asciidoc itself was 'in maintenance mode for existing users'. I've tried
asciidoctor but never got it to the point where I was happy with the
results. Having two tools using the same nominal format doesn't seem
like a great idea to me.

It's also clear from my hacking in asciidoc that docbook is the expected
target for that tool. I've managed to make direct HTML output usable,
but LaTeX doesn't work at all. Something which focuses on direct HTML
(and ePub) output would be pretty nice.

> An alternative we haven't really looked at yet is ReStructuredText (or
> "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is
> YA simple markup scheme, remarkably similar to Markdown or Asciidoc;
> Sphinx is a fairly sophisticated documentation system that uses RST.

I've installed debian's python3-sphinx package; it looks like it doesn't
have a huge dependency chain below it, which is a nice change.

I translated a fairly long document from asciidoc to rst using pandoc by
using the docbook output from asciidoc -- pandoc doesn't have a native
asciidoc reader, only a writer. The result didn't totally suck, although
I haven't messed with fixing the css or using a different theme at all.

http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html

I installed the sphinxcontrib.fulltoc extension so that the whole TOC
was visible from each section; this made navigating a lot easier. Having
search included (if you have javascript) seems like a nice feature.

> Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain
> requirements there.

Having functional native latex output means that even PDF generation is
lighterweight though.

> It produces integrated, multi-file HTML natively,
> with a TOC, an index, cross-file cross references, and more. It can make
> things like function indexes. It claims output in epub, docbook, and man
> (I've not yet messed with those). The path to PDF is via latex; clearly
> the docbook path could be used too.

I've tried epub and latex backends; epub seems just fine (it's just
html, after all). LaTeX works, and generates functional PDF, but I'm
going to have to spend a bunch of time making it looks nice.

http://keithp.com/~keithp/altusmetrum-sphinx/AltusMetrum.pdf

> So can we discuss? I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take. What do you all think?

Having spent the afternoon playing with it, I'm definitely
impressed. I've spent a ton of time getting asciidoc to generate html
and pdf that I can tolerate; far too much of that involved hacking XML
files related to the docbook backend.

Pros

* Credible HTML output without docbook

* Credible PDF output without docbook.

* Constructs a unified set of documents across
multiple files.

* Written in Python (2 or 3)

* PanDoc already supports rst for both input and output. So, if we get
bored with RST, we've got a way out.

Cons

* Table formatting doesn't seem as sophisticated as asciidoc

Questions

* Conditional text appears to be harder to manage (I haven't managed to
make it work at all).

* Takes over a directory making building more than one
document in a directory hard/impossible? The config file must be
named 'conf.py'?

--
-keith

[Daniel Vetter]

One concern/open I have for pro/cons are the hyperlinks from kerneldoc
comments. Currently we have the postproc hack, iirc Jani's patches
generated links native when extracting the kerneldoc. What's the
solution with spinx?

The other one is graphs - Keith showed me some neat stuff that
asciidoc can do, and I definitely wanted to integrate something like
that as a follow-up into the kerneldoc toolchain. Often a diagram is a
lot more helpful than lots of words. Can sphinx gives us that too?

Wrt reformatting: I'm not going to like it, but I hope that with a bit
of sed we can fix up any of the asciidoc comments we have already
easily - right now we don't (yet) use much of the more sophisticated
markup yet. So much better to change now than 1 year down the road.

Cheers, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[Keith Packard]

Daniel Vetter <daniel...@ffwll.ch> writes:

> The other one is graphs - Keith showed me some neat stuff that
> asciidoc can do, and I definitely wanted to integrate something like
> that as a follow-up into the kerneldoc toolchain. Often a diagram is a
> lot more helpful than lots of words. Can sphinx gives us that too?

.. graphviz::

digraph foo {
"bar" -> "baz";
}

Even better than asciidoc -- svg output is supported in both html and
pdf (when using rst2pdf). I had to hack asciidoc to add support for svg
output when using docbook.

> Wrt reformatting: I'm not going to like it, but I hope that with a bit
> of sed we can fix up any of the asciidoc comments we have already
> easily - right now we don't (yet) use much of the more sophisticated
> markup yet. So much better to change now than 1 year down the road.

I used pandoc on the docbook output from asciidoc to get a 100 page
document converted here. It wasn't perfect -- all of the internal links
were busted, and labels for tables were mis-positioned. It might be that
a few minor fixes to pandoc could be done to add 'sphinx'-specific rst
support that could fix this?

I spent (too much) time yesterday playing with sphinx and generated a
new html theme. Here's the result:

http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html

Here's the PDF output from rst2pdf, a python-based PDF output which
doesn't use docbook *or* latex:

http://keithp.com/~keithp/altusmetrum-sphinx/Altus%20Metrum.pdf

I need to spend some quality time building my own PDF theme; the default
provided by rst2pdf isn't great. It does, however, use fontconfig, so
switching fonts is *way* easier than with docbook...

There's currently an incompatibility between the rst2pdf and sphnix
packages in debian (and upstream) which I hacked around to generate that
output, but otherwise I'm using packaged bits.

So, another pro for sphinx appears to be native PDF generation...

--
-keith

[Jani Nikula]

On Sat, 13 Feb 2016, Jonathan Corbet <cor...@lwn.net> wrote:
> So can we discuss? I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take. What do you all think?

FWIW I was in favor of reStructuredText to begin with, but decided I'd
rather have any lightweight markup than a bikeshed fest over which
one. Switching is a non-issue to me.

It does seem like the tools available for each markup carry more weight
than the markup itself, as long as there aren't any huge gaps. Sphinx
seems like a reasonable, actively developed choice.

However I didn't think Sphinx could produce docbook, and a quick search
doesn't convince me otherwise. Do you have some links to back this up?
Would the lack of docbook be a showstopper? (Of course, the pandoc
swiss-army knife can handle rst->docbook if needed.)

Sphinx might offer a way to drop docproc through the extension
mechanism, without resorting to the "separate-file approach". It might
be a more sensible approach as a whole.

I'll have a more in-depth look into this.

BR,
Jani.


--
Jani Nikula, Intel Open Source Technology Center

[Jonathan Corbet]

On Tue, 16 Feb 2016 10:25:49 +0200
Jani Nikula <jani....@intel.com> wrote:

> However I didn't think Sphinx could produce docbook, and a quick search
> doesn't convince me otherwise. Do you have some links to back this up?

Somehow I was really sure of it, but I'm not finding it now. There is an
extension out there, but it warns about being a "work in progress," so
I'm not sure we can count it.

Whether this is a show-stopper is indeed a good question. I doubt many
people wanted the DocBook for its own sake, it's a matter of where you
can go from there. But yes, it would be good to be sure on this point.

> Sphinx might offer a way to drop docproc through the extension
> mechanism, without resorting to the "separate-file approach". It might
> be a more sensible approach as a whole.

There's a certain elegance to it that I like, but it is an idea that
needs to actually be demonstrated. It could also come later on, though,
with the docproc or include mechanisms used for now.

Thanks,

jon

[Jani Nikula]

On Tue, 16 Feb 2016, Jonathan Corbet <cor...@lwn.net> wrote:
> Whether this is a show-stopper is indeed a good question. I doubt many
> people wanted the DocBook for its own sake, it's a matter of where you
> can go from there. But yes, it would be good to be sure on this point.

So the question is, are HTML, latex (for pdf and presumably ps), epub,
texinfo, and man pages enough?

My subjective opinion is they are enough, and whoever wants more can
make it work for them using pandoc. It's not like we had this great
publishing framework before that we'd be breaking. Quite the opposite,
and it would be awesome to have a robust set of outputs actually working
for the majority of people.

> There's a certain elegance to it that I like, but it is an idea that
> needs to actually be demonstrated. It could also come later on, though,
> with the docproc or include mechanisms used for now.

Oh, totally agreed, I was just musing on potential follow-up work.

[Keith Packard]

Jani Nikula <jani....@intel.com> writes:

> However I didn't think Sphinx could produce docbook, and a quick search
> doesn't convince me otherwise. Do you have some links to back this up?
> Would the lack of docbook be a showstopper? (Of course, the pandoc
> swiss-army knife can handle rst->docbook if needed.)

A quick search yields:

https://github.com/HolgerPeters/sphinxcontrib-docbook

which appears to provide docbook output for sphinx, but I haven't tested
this at all.

Of course, docbook is just a means to an end -- the question should
probably be what final output formats are actually required and whether
the tools generate those in a reasonable fashion.

--
-keith

[Jonathan Corbet]

On Tue, 16 Feb 2016 11:13:29 -0800
Keith Packard <kei...@keithp.com> wrote:

> https://github.com/HolgerPeters/sphinxcontrib-docbook
>
> which appears to provide docbook output for sphinx, but I haven't tested
> this at all.

Yup, that's the one I found, the one that says "Very much work in
progress". It looks like that work stopped almost a year ago.

> Of course, docbook is just a means to an end -- the question should
> probably be what final output formats are actually required and whether
> the tools generate those in a reasonable fashion.

Indeed, I doubt many people want the DocBook itself.

jon

[Keith Packard]

Jonathan Corbet <cor...@lwn.net> writes:

> Indeed, I doubt many people want the DocBook itself.

Might be nice to actually have a set of requirements before anyone tries
to select a suitable system then :-)

Here's my current set:

asciidoc sphinx

html via docbook native
native (kinda)
pdf via docbook via rst2pdf
epub via docbook? native
man ? native?

--
-keith

[Jonathan Corbet]

On Sun, 14 Feb 2016 13:27:04 +0100
Daniel Vetter <daniel...@ffwll.ch> wrote:

> One concern/open I have for pro/cons are the hyperlinks from kerneldoc
> comments. Currently we have the postproc hack, iirc Jani's patches
> generated links native when extracting the kerneldoc. What's the
> solution with spinx?

So I've been trying to figure out what this refers to. Is this the
cross-reference links within the document? When I did my sphinx hack it
used a technique that, shall we say, strongly resembles what Jani's
patches did. One difference is that Sphinx has the concept of
"functions" built into it, so I use function references for those.

If you mean links to the wider world, I'm not sure I see that in Jani's
patches. It's going to be easily managed in whatever markup language we
use, though.

jon

[Daniel Vetter]

On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet <cor...@lwn.net> wrote:
> On Sun, 14 Feb 2016 13:27:04 +0100
> Daniel Vetter <daniel...@ffwll.ch> wrote:
>
>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc
>> comments. Currently we have the postproc hack, iirc Jani's patches
>> generated links native when extracting the kerneldoc. What's the
>> solution with spinx?
>
> So I've been trying to figure out what this refers to. Is this the
> cross-reference links within the document? When I did my sphinx hack it
> used a technique that, shall we say, strongly resembles what Jani's
> patches did. One difference is that Sphinx has the concept of
> "functions" built into it, so I use function references for those.

That's what I meant. As long as I can type in stuff like func(),
&struct and similar and get a link for it automatically (plus anywhere
else in the templated stuff for function headers) I'm really happy.
-Daniel

[Jani Nikula]

On Thu, 18 Feb 2016, Daniel Vetter <daniel...@ffwll.ch> wrote:
> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet <cor...@lwn.net> wrote:
>> On Sun, 14 Feb 2016 13:27:04 +0100
>> Daniel Vetter <daniel...@ffwll.ch> wrote:
>>
>>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc
>>> comments. Currently we have the postproc hack, iirc Jani's patches
>>> generated links native when extracting the kerneldoc. What's the
>>> solution with spinx?
>>
>> So I've been trying to figure out what this refers to. Is this the
>> cross-reference links within the document? When I did my sphinx hack it
>> used a technique that, shall we say, strongly resembles what Jani's
>> patches did. One difference is that Sphinx has the concept of
>> "functions" built into it, so I use function references for those.
>
> That's what I meant. As long as I can type in stuff like func(),
> &struct and similar and get a link for it automatically (plus anywhere
> else in the templated stuff for function headers) I'm really happy.

I think that could be made to work in rst just as well as
asciidoc. Which is to say, kernel-doc may generate broken refs in both,
since it doesn't know if the link target exists outside of the
file. Also, in theory, it's possible to generate non-unique targets in
the end result if there are same named enums, structs, static functions
etc. but I think that's less of a problem. Asciidoc just ignores these
issues, I don't know what sphinx does.

I think some of that could be alleviated by making the kernel-doc
inclusion a directive through a sphinx extension. It could at the very
least provide informative error messages. But that's distant future.

Worth noting is that, AFAICT, in all of the proposals, including the
original where kernel-doc produces docbook, this autoreferencing only
works within parts processed by kernel-doc. Not in the template
documents themselves. (You can still use the markup's more verbose cross
referencing keywords.)

[Daniel Vetter]

It works everywhere, even in the docbook template, as long as you mark
it up correctly. Which in docbook means <function>func</function>.
That's because it's a post-proc path over the entire doc. But then the
entire point here is to move the overview sections all into kerneldoc,
so making the links in the templates more verbose shouldn't be a big
deal.

[Mauro Carvalho Chehab]

Em Thu, 18 Feb 2016 10:24:04 +0100
Daniel Vetter <daniel...@ffwll.ch> escreveu:

For simple documents like the one produced by kernel-doc, I guess
all markup languages would work equally.

The problem is for complex documents like the media kAPI one, where
the document was written to produce a book. So, it uses some complex
features found at DocBook. One of such features we use extensively
is the capability of having a table with per-line columns. This way,
we can produce things like:

V4L2_CID_COLOR_KILLER boolean Enable the color killer (i. e. force a black & white image in case of a weak video signal).
V4L2_CID_COLORFX enum Selects a color effect. The following values are defined:
V4L2_COLORFX_NONE Color effect is disabled.
V4L2_COLORFX_ANTIQUE An aging (old photo) effect.
V4L2_COLORFX_ART_FREEZE Frost color effect.

In the above example, we have a main 3 columns table, and we embed
a 2 columns table at the third field of V4L2_CID_COLORFX to represent
possible values for this menu control.

See https://linuxtv.org/downloads/v4l-dvb-apis/control.html for the
complete output of it.

This is used extensively inside the media DocBook, and properly
supporting it is one of our major concerns.

Are there any way to represent those things with the markup
languages currently being analyzed?

Converting those tables will likely require manual work, as I don't
think automatic tools will properly handle it, specially since we
use some DocBook macros to help creating such tables.

--
Thanks,
Mauro

[Jani Nikula]

Since I've let myself be told that asciidoc handles tables better than
reStructuredText, I tested this a bit with the presumably inferior one.

rst has two table types, simple tables and grid tables [1]. It seems
like grid tables can do pretty much anything, but they can be cumbersome
to work with. So I tried to check what can be done with simple tables.

Here's a sample, converted using rst2html (Sphinx will be prettier, but
rst2html works for simple things like this):

https://people.freedesktop.org/~jani/v4l-table-within-table.rst
https://people.freedesktop.org/~jani/v4l-table-within-table.html

Rather than using nested tables, you might want to consider using
definition lists within tables:

https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst
https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html

You be the judge, but I think this is workable.

BR,
Jani.


[1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#tables

[Hans Verkuil]

On 02/18/16 13:04, Mauro Carvalho Chehab wrote:
> Em Thu, 18 Feb 2016 13:23:37 +0200
> Jani Nikula <jani....@intel.com> escreveu:

> Yes, this would work. Can we remove the border from the main table?
> I guess it would be nicer.

>
>>
>> Rather than using nested tables, you might want to consider using
>> definition lists within tables:
>>
>> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst
>> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html
>>
>> You be the judge, but I think this is workable.
>

> It is workable, but I guess nested tables produced a better result.
>
> I did myself a test with nested tables with asciidoc too:
>
> https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html
> https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii
>
> With looks very decent to me.

It does, except for the vertical alignment of the third column (at least when viewed
with google chrome).

Hans

>
> I had to manually add the nested table, as pandoc conversion sent the
> DocBook's nested table to /dev/null.
>
> Thanks,
> Mauro
>

[Mauro Carvalho Chehab]

Em Thu, 18 Feb 2016 13:23:37 +0200
Jani Nikula <jani....@intel.com> escreveu:

Yes, this would work. Can we remove the border from the main table?
I guess it would be nicer.

>

> Rather than using nested tables, you might want to consider using
> definition lists within tables:
>
> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.rst
> https://people.freedesktop.org/~jani/v4l-definition-list-within-table.html
>
> You be the judge, but I think this is workable.

It is workable, but I guess nested tables produced a better result.

I did myself a test with nested tables with asciidoc too:

https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html
https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii

With looks very decent to me.

[Mauro Carvalho Chehab]

Em Thu, 18 Feb 2016 13:07:03 +0100
Hans Verkuil <hver...@xs4all.nl> escreveu:

Not sure what you mean. Here, it looks fine on both Firefox and Chrome,
except that the second colum size could be smaller. If this is what
you're meaning this can be fixed by changing the second line from:

[width="100%",cols="2,1,10a",options="header",frame="none", grid="none"]

to:

[width="100%",cols="3,1,30a",options="header",frame="none", grid="none"]

With regards to ReStructured Text, I've no idea how to control the
format of a table in order to do things like hiding the borders and
changing the column spacing.

So, at least on a first glance, asciidoc seems to fit better.

Thanks,
Mauro

[Jonathan Corbet]

On Thu, 18 Feb 2016 10:24:04 +0100
Daniel Vetter <daniel...@ffwll.ch> wrote:

> > Worth noting is that, AFAICT, in all of the proposals, including the
> > original where kernel-doc produces docbook, this autoreferencing only
> > works within parts processed by kernel-doc. Not in the template
> > documents themselves. (You can still use the markup's more verbose cross
> > referencing keywords.)
>
> It works everywhere, even in the docbook template, as long as you mark
> it up correctly. Which in docbook means <function>func</function>.

In Sphinx that's :func:`function_name`. Not a thing of beauty, but it
works.

jon

[Patrick Boettcher]

On Thu, 18 Feb 2016 10:44:34 -0200
Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:
> > > It is workable, but I guess nested tables produced a better
> > > result.
> > >
> > > I did myself a test with nested tables with asciidoc too:
> > >
> > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.html
> > > https://mchehab.fedorapeople.org/media-kabi-docs-test/pandoc_asciidoc/table.ascii
> > >
> > > With looks very decent to me.
> >
> > It does, except for the vertical alignment of the third column (at
> > least when viewed with google chrome).
>
> Not sure what you mean. Here, it looks fine on both Firefox and
> Chrome, except that the second colum size could be smaller. If this
> is what you're meaning this can be fixed by changing the second line
> from:

I think Hans' problem (I see it as well) is coming from css-style of
"paragraph" which is:

 margin-top: 0.5em;
 margin-bottom: 0.5em;

This makes the third column non-vertical-aligned

--
Patrick

[Daniel Vetter]

Compared to <function>function_name</function> already an improvement.
I can very much live with this ;-)

[Jani Nikula]

On Sat, 13 Feb 2016, Jonathan Corbet <cor...@lwn.net> wrote:
> So can we discuss? I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take. What do you all think?

This stalled a bit, but the waters are still muddy...

Is the Sphinx/reStructuredText table support adequate for media/v4l
documentation?

Are the Sphinx output formats adequate in general? Specifically, is the
lack of DocBook support, and the flexibility it provides, a blocker?

Otherwise, I think Sphinx is promising.

Jon, I think we need a roll of dice, err, a well-thought-out decision
from the maintainer to go with one or the other, so we can make some
real progress.

BR,
Jani.

[Jonathan Corbet]

On Thu, 03 Mar 2016 16:03:14 +0200
Jani Nikula <jani....@intel.com> wrote:

> This stalled a bit, but the waters are still muddy...

I've been dealing with real-world obnoxiousness, something which won't
come to an immediate end, unfortunately. But I have been taking some time
to mess with things, and hope to have some more POC patches to send out
soon.

> Is the Sphinx/reStructuredText table support adequate for media/v4l
> documentation?

That's perhaps the biggest question. My sense is "yes", but this needs a
bit more assurance than that.

> Are the Sphinx output formats adequate in general? Specifically, is the
> lack of DocBook support, and the flexibility it provides, a blocker?

DocBook is a means to an end; nobody really wants DocBook itself as far
as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
reasonable output, and, with some effort, we could probably get really
nice output. HTML and EPUB are easily covered, still haven't really played
around with man pages yet. And there's LaTeX if we really need it. I
kind of think we're covered there, unless I've missed something?

> Otherwise, I think Sphinx is promising.
>
> Jon, I think we need a roll of dice, err, a well-thought-out decision
> from the maintainer to go with one or the other, so we can make some
> real progress.

My inclination at the moment is very much in the Sphinx direction. I had
some vague thoughts of pushing a throwaway experimental directory with a
couple of docs for 4.6 that would just let people play with it easily;
then we'd see how many screams we get. We'll see if the world lets me get
there.

Thanks,

jon

[One Thousand Gnomes]

> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell.

We only have docbook because it was the tool of choice rather a lot of
years ago to then get useful output formats. It was just inherited when
borrowed the original scripts from Gnome/Gtk. It's still the most
effective way IMHO of building big structured documents out of the kernel.

The Gtk people long ago rewrote the original document script into a real
tool so they have some different and maintained tools that are close to
equivalent and already have some markdown support. Before we go off and
re-invent the wheel it might be worth just borrowing their wheel and
tweaking it as needed ? In particular they can generate help indexes so
that the entire output becomes nicely browsable with an HTML based help
browser.

Alan

[Jonathan Corbet]

On Thu, 3 Mar 2016 14:34:25 +0000
One Thousand Gnomes <gno...@lxorguk.ukuu.org.uk> wrote:

> We only have docbook because it was the tool of choice rather a lot of
> years ago to then get useful output formats. It was just inherited when
> borrowed the original scripts from Gnome/Gtk. It's still the most
> effective way IMHO of building big structured documents out of the kernel.

...except that we haven't used it that way. Instead, we make a whole
bunch of smaller, partially structured document silos.

> The Gtk people long ago rewrote the original document script into a real
> tool so they have some different and maintained tools that are close to
> equivalent and already have some markdown support. Before we go off and
> re-invent the wheel it might be worth just borrowing their wheel and
> tweaking it as needed ? In particular they can generate help indexes so
> that the entire output becomes nicely browsable with an HTML based help
> browser.

Well, not inventing the wheel was kind of the motivation behind much of
this effort; I got kind of worried watching us trying to cobble more
functionality into our existing house-of-cards documentation system.

Sphinx is a well-established, heavily used, and well supported system;
using it would not be an exercise in wheel reinvention. As far as I can
tell, it does everything we need (with some open questions about table
support), lets us drop the whole DocBook toolchain dependency, and move to
a much better-supported setup than we have now. Plus we get much nicer
output, index generation, cross-references between documents, and the
ability to write documents in a lightweight markup language. Seems like a
win.

I assume you're referring to gtk-doc? It's web page
(http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
setup and use"; they recommend looking at Doxygen instead. So I guess I'm
not really sure what it offers that merits throwing another option into
the mix now? What am I missing?

Thanks,

jon

[Daniel Vetter]

On Thu, Mar 3, 2016 at 4:17 PM, Jonathan Corbet <cor...@lwn.net> wrote:
> I assume you're referring to gtk-doc? It's web page
> (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
> setup and use"; they recommend looking at Doxygen instead. So I guess I'm
> not really sure what it offers that merits throwing another option into
> the mix now? What am I missing?

We use gtk-doc for the i915 testcase and tooling repo in userspace
(intel-gpu-tools). The setup is somewhat arcane (some build-fu that is
fumbly, and xml files to tie everything together). But it looks pretty
and works well otherwise. It should be at
https://01.org/linuxgraphics/gfx-docs/igt/ but our autobuilder seems
to be screwed up right now.

Of course I considered it as an option, but like doxygen it has it's
own strong opinion about how in-code comments should look like, and
those differ from kerneldoc syntax. Beyond that I don't really see
benefits over any of the solutions proposed here already (either
sphinx or rst or horror! even the hackfest I still carry around in
drm-intel.git branches).

Btw for igt we went with gtkdoc over docygen because a few people on
our team had "doxygen only over my corpse" level kind of strong
opinions. Everyone just loves their own color choice for this bikeshed

[Mauro Carvalho Chehab]

Em Thu, 03 Mar 2016 07:13:05 -0700
Jonathan Corbet <cor...@lwn.net> escreveu:

> On Thu, 03 Mar 2016 16:03:14 +0200
> Jani Nikula <jani....@intel.com> wrote:
>
> > This stalled a bit, but the waters are still muddy...
>
> I've been dealing with real-world obnoxiousness, something which won't
> come to an immediate end, unfortunately. But I have been taking some time
> to mess with things, and hope to have some more POC patches to send out
> soon.
>
> > Is the Sphinx/reStructuredText table support adequate for media/v4l
> > documentation?
>
> That's perhaps the biggest question. My sense is "yes", but this needs a
> bit more assurance than that.

On my tests, Sphinix seemed too limited to format tables. Asciidoc
produced an output that worked better.

Please notice that we tried to convert only one type of table. The result
with RST was not beautiful, but worked.

However, we use tables also to show how bits appear at the video formats,
like the tables at:
https://linuxtv.org/downloads/v4l-dvb-apis/subdev.html#v4l2-mbus-format

For those tables to look nice, we should be able to remove borders and grids
from the table. I was unable to find a way to control the tables format
with RST to do things like grid/border removal.

> > Are the Sphinx output formats adequate in general? Specifically, is the
> > lack of DocBook support, and the flexibility it provides, a blocker?
>
> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
> reasonable output, and, with some effort, we could probably get really
> nice output. HTML and EPUB are easily covered, still haven't really played
> around with man pages yet. And there's LaTeX if we really need it. I
> kind of think we're covered there, unless I've missed something?
>
> > Otherwise, I think Sphinx is promising.
> >
> > Jon, I think we need a roll of dice, err, a well-thought-out decision
> > from the maintainer to go with one or the other, so we can make some
> > real progress.
>
> My inclination at the moment is very much in the Sphinx direction. I had
> some vague thoughts of pushing a throwaway experimental directory with a
> couple of docs for 4.6 that would just let people play with it easily;
> then we'd see how many screams we get. We'll see if the world lets me get
> there.

I'm not against having a staging/Documentation for us to play with,
provided, of course, that whatever tool chosen would allow converting
what we have today.

Regards,
Mauro

[Keith Packard]

Mauro Carvalho Chehab <mch...@osg.samsung.com> writes:

> On my tests, Sphinix seemed too limited to format tables. Asciidoc
> produced an output that worked better.

Yes, asciidoc has much more flexibility in table formatting, including
the ability to control text layout within cells and full control over
borders.

However, I think asciidoc has two serious problems:

1) the python version (asciidoc) appears to have been abandoned in
favor of the ruby version.

2) It really is just a docbook pre-processor. Native html/latex output
is poorly supported at best, and exposes only a small subset of the
full capabilities of the input language.

As such, we would have to commit to using the ruby version and either
committing to fixing the native html output backend or continuing to use
the rest of the docbook toolchain.

We could insist on using the python version, of course. I spent a bit of
time hacking that up to add 'real' support for a table-of-contents in
the native HTML backend and it looks like getting those changes
upstreamed would be reasonably straightforward. However, we'd end up
'owning' the code, and I'm not sure we want to.

--
-keith

[Mauro Carvalho Chehab]

Em Thu, 03 Mar 2016 15:23:23 -0800
Keith Packard <kei...@keithp.com> escreveu:

I'm a way more concerned about using a tool that fulfill our needs
than to look for something that won't use the docbook toolchain or
require to install ruby.

In the case of Docbook, we know it works and we know already its
issues. Please correct me if I'm wrong, but the big problem we
have is not due to the DocBook toolchain, but due to the lack of
features at the kernel-doc script. Also, xmlto is already installed
by the ones that build the kernel docs. So, keeping use it won't
require to install a weird toolchain by hand.

So, to be frank, it doesn't scary me to use either pyhton or
ruby script + docbook.

Of course, having to own the code has a cost that should be evaluated.

If, on the other hand, we decide to use RST, we'll very likely need to
patch it to fulfill our needs in order to add proper table support.
I've no idea how easy/difficult would be to do that, nor if Sphinx
upstream would accept such changes.

So, at the end of the day, we may end by having to carry on our own
version of Sphinx inside our tree, with doesn't sound good, specially
since it is not just a script, but a package with hundreds of
files.

Thanks,
Mauro

[Russel Winder]

On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
>
[…]

> However, I think asciidoc has two serious problems:
>
>   1) the python version (asciidoc) appears to have been abandoned in
>      favor of the ruby version. 

This is I think true, however the Java-based tool chain Asciidoctor is
I believe the standard bearer for ASCIIdoc these days, albeit called
ASCIIdoctor.

>   2) It really is just a docbook pre-processor. Native html/latex
> output
>      is poorly supported at best, and exposes only a small subset of
> the
>      full capabilities of the input language.

This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as
to use a sane :-) markup language rather than XML (XML is a notation
for consenting computers only), but the current ASCIIDoctor toolchain
deals very well in direct HTML and PDF generation, without needing a
DocBook/XML toolchain. 

> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to
> use
> the rest of the docbook toolchain.

Or trial the JVM-based ASCIIdoctor which is what the projects I am
involved with chose to use. Perhaps as an example I can give you http:/
/gpars.website (it's a redirector) all the HTML and PDF is generated
from ASCIIDoc source using ASCIIDoctor driven with a Gradle build
system. This is still very much a work in progress (by Jim Northrop,
not me currently), but I like it.

> We could insist on using the python version, of course. I spent a bit
> of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

If the Python version is really not being maintained, I would suggest
that unless you want to take over the project and be it's maintainer,
you would be better advised to use a different version.

-- Russel.=============================================================================Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel...@ekiga.net41 Buckmaster Road    m: +44 7770 465 077   xmpp: rus...@winder.org.ukLondon SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

[Jani Nikula]

On Fri, 04 Mar 2016, Russel Winder <rus...@winder.org.uk> wrote:
> On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
>>   1) the python version (asciidoc) appears to have been abandoned in
>>      favor of the ruby version. 
>
> This is I think true, however the Java-based tool chain Asciidoctor is
> I believe the standard bearer for ASCIIdoc these days, albeit called
> ASCIIdoctor.

If we're talking about the same asciidoctor (http://asciidoctor.org/)
it's written in ruby but you can apparently run it in JVM using
JRuby. Calling it Java-based is misleading.

[Jani Nikula]

I think you meant that to be the other way round, or I fail at parsing
you. ;)

> In the case of Docbook, we know it works and we know already its
> issues. Please correct me if I'm wrong, but the big problem we
> have is not due to the DocBook toolchain, but due to the lack of
> features at the kernel-doc script. Also, xmlto is already installed
> by the ones that build the kernel docs. So, keeping use it won't
> require to install a weird toolchain by hand.

I think kernel-doc is just a small part of the puzzle. It's a problem,
but a small one at that, and we've already made it output asciidoc, rst
and docbook as part of this exercise. For real, as in code, not as in
talk.

The reasons I'm involved in this is that I want to make writing
documentation and rich kernel-doc comments easier (using lightweight
markup) and I want to make building the documentation easier (using a
straightforward toolchain with not too many dependencies). I'm hoping
the former makes writing documentation more attractive and the latter
keeps the documentation and the toolchain in a better shape through
having more people actually build the documentation.

IMHO docbook is problematic because the toolchain gets too long and
fragile. You need plenty of tools installed to build the documentation,
it's fussy to get working, and people just won't. Like code,
documentation bitrots too when it's not used. The documentation build is
broken too often. Debugging formatting issues through the entire
pipeline gets hard; I already faced some of this when playing with the
kernel-doc->asciidoc->docbook->html chain.

In short, I don't think the docbook toolchain fills all of our needs
either.

> So, to be frank, it doesn't scary me to use either pyhton or
> ruby script + docbook.
>
> Of course, having to own the code has a cost that should be evaluated.
>
> If, on the other hand, we decide to use RST, we'll very likely need to
> patch it to fulfill our needs in order to add proper table support.
> I've no idea how easy/difficult would be to do that, nor if Sphinx
> upstream would accept such changes.
>
> So, at the end of the day, we may end by having to carry on our own
> version of Sphinx inside our tree, with doesn't sound good, specially
> since it is not just a script, but a package with hundreds of
> files.

If we end up having to modify Sphinx, it has a powerful extension
mechanism for this. We wouldn't have to worry about getting it merged to
Sphinx upstream, and we wouldn't have to carry a local version of all of
Sphinx. (In fact, the extension mechanism provides a future path for
doing kernel-doc within Sphinx instead of as a preprocessing step.)

I know none of this alleviates your concerns with table supports right
now. I'll try to have a look at that a bit more.

[Johannes Stezenbach]

FWIW, I think table formatting in Sphinx works via style sheets.
The mechanism is documented in the Python docutils docs that
Sphinx is built upon.
Basically you use the "class" or "role" directive and define
the corresponding CSS or LaTeX (or rst2pdf) style.

Here is one example (using a custom "cssclass" role):
https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html

Directives (especially role and class):
http://www.sphinx-doc.org/en/stable/rest.html#directives

LaTeX styling:
http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles


HTH,
Johannes

[Mauro Carvalho Chehab]

Em Fri, 04 Mar 2016 10:29:08 +0200
Jani Nikula <jani....@intel.com> escreveu:

I mean: I'm a way more concerned about using a tool that fulfill our
needs than on toolchain it uses.

>
> > In the case of Docbook, we know it works and we know already its
> > issues. Please correct me if I'm wrong, but the big problem we
> > have is not due to the DocBook toolchain, but due to the lack of
> > features at the kernel-doc script. Also, xmlto is already installed
> > by the ones that build the kernel docs. So, keeping use it won't
> > require to install a weird toolchain by hand.
>
> I think kernel-doc is just a small part of the puzzle. It's a problem,
> but a small one at that, and we've already made it output asciidoc, rst
> and docbook as part of this exercise. For real, as in code, not as in
> talk.
>
> The reasons I'm involved in this is that I want to make writing
> documentation and rich kernel-doc comments easier (using lightweight
> markup) and I want to make building the documentation easier (using a
> straightforward toolchain with not too many dependencies). I'm hoping
> the former makes writing documentation more attractive and the latter
> keeps the documentation and the toolchain in a better shape through
> having more people actually build the documentation.

I don't think the toolchain is a problem. We don't attract too many
people because developers don't like writing documentation.

Ok, using a markup language can be easier than writing DocBook
tags directly, but people usually don't even add C comments on
the code they submit.

> IMHO docbook is problematic because the toolchain gets too long and
> fragile. You need plenty of tools installed to build the documentation,
> it's fussy to get working, and people just won't. Like code,
> documentation bitrots too when it's not used.

On most distros, a single command installs all that it is needed.

> The documentation build is broken too often.

This is indeed a problem, but the way I solved this at the media
subsystem is that I rebuild the documentation every time a media
file either at Documentation/Docbook or included at device-drivers.tmpl
is touched.

If the script produces error, I nack the patch.

So, IMHO, this is not a toolchain fault, but the lack of a process.

The kernel build robot is now producing e-mails when the documentation
has new troubles, so I guess this will help a lot to avoid adding
new documentation breakages.

> Debugging formatting issues through the entire
> pipeline gets hard; I already faced some of this when playing with the
> kernel-doc->asciidoc->docbook->html chain.
>
> In short, I don't think the docbook toolchain fills all of our needs
> either.
>
> > So, to be frank, it doesn't scary me to use either pyhton or
> > ruby script + docbook.
> >
> > Of course, having to own the code has a cost that should be evaluated.
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.
>
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)

That's indeed a good news.

>
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

I created a PoC tree with a few usecases taken from the V4L2 uAPI
documentation:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Perhaps you could help to fix the issues there:

1) We want borderless tables, on both PDF and HTML outputs, for
the table at:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/tree/v4l-table-within-table.rst

2) The tables at packed-rgb.rst are not created:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/diff/packed-rgb.rst

It complains that the table there is malformed:
packed-rgb.rst:12: ERROR: Malformed table.

3) I tried to use a .. cssclass, as Johannes suggested, but
I was not able to include the CSS file. I suspect that this is
easy to fix, but I want to see if the cssclass will also work for
the pdf output as well.

4) It seems that it can't produce nested tables in pdf:

Markup is unsupported in LaTeX:
v4l-table-within-table:: nested tables are not yet implemented.
Makefile:115: recipe for target 'latexpdf' failed

Can you help solving those issues?

Thanks,
Mauro

[Johannes Stezenbach]

On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
>
> 3) I tried to use a .. cssclass, as Johannes suggested, but
> I was not able to include the CSS file. I suspect that this is
> easy to fix, but I want to see if the cssclass will also work for
> the pdf output as well.

"cssclass" was (I think) a custom role defined in the example,
unless you also have defined a custom role you can use plain "class".
I have not looked deeper into the theming and template stuff.

> 4) It seems that it can't produce nested tables in pdf:
>
> Markup is unsupported in LaTeX:
> v4l-table-within-table:: nested tables are not yet implemented.
> Makefile:115: recipe for target 'latexpdf' failed

This:
http://www.sphinx-doc.org/en/stable/markup/misc.html#tables

suggests you need to add the tabularcolumns directive
for complex tables.

BTW, as an alternative to the ASCII-art input
there is also support for CSV and list tables:
http://docutils.sourceforge.net/docs/ref/rst/directives.html#table


Johannes

[Mauro Carvalho Chehab]

Em Fri, 04 Mar 2016 15:09:09 +0100
Johannes Stezenbach <j...@linuxtv.org> escreveu:

> On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> >
> > 3) I tried to use a .. cssclass, as Johannes suggested, but
> > I was not able to include the CSS file. I suspect that this is
> > easy to fix, but I want to see if the cssclass will also work for
> > the pdf output as well.
>
> "cssclass" was (I think) a custom role defined in the example,
> unless you also have defined a custom role you can use plain "class".
> I have not looked deeper into the theming and template stuff.

Well, it accepted cssclass for html (well, it didn't find the
templates - so I guess it is just me failing to understand how to tell
sphinx to get the stylesheet), but it rejects it for latexPDF.

>
> > 4) It seems that it can't produce nested tables in pdf:
> >
> > Markup is unsupported in LaTeX:
> > v4l-table-within-table:: nested tables are not yet implemented.
> > Makefile:115: recipe for target 'latexpdf' failed
>
> This:
> http://www.sphinx-doc.org/en/stable/markup/misc.html#tables
>
> suggests you need to add the tabularcolumns directive
> for complex tables.
>
> BTW, as an alternative to the ASCII-art input
> there is also support for CSV and list tables:
> http://docutils.sourceforge.net/docs/ref/rst/directives.html#table

I converted one of the big tables to CSV. At least now it recognized
it as a table. Yet, the table was very badly formated:
https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html

This is how this table should look like:
https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html

Also, as this table has merged cells at the legend. I've no idea how
to tell sphinx to do that on csv format.

The RST files are on this git tree:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Regards,
Mauro

[Johannes Stezenbach]

On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
>
> I converted one of the big tables to CSV. At least now it recognized
> it as a table. Yet, the table was very badly formated:
> https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
>
> This is how this table should look like:
> https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
>
> Also, as this table has merged cells at the legend. I've no idea how
> to tell sphinx to do that on csv format.
>
> The RST files are on this git tree:
> https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Yeah, seems it can't do merged cells in csv. Attached patch converts it
back to grid table format and fixes the table definition.
The html output looks usable, but clearly it is no fun to
work with tables in Sphinx.

Sphinx' latex writer can't handle nested tables, though.
Python's docutils rst2latex can, but that doesn't help here.
rst2pdf also supports it. But I have doubts such a large
table would render OK in pdf without using landscape orientation.
I have not tried because I used python3-sphinx but rst2pdf
is only availble for Python2 in Debian so it does not integrate
with Sphinx.


Johannes

[Jonathan Corbet]

On Thu, 03 Mar 2016 16:03:14 +0200
Jani Nikula <jani....@intel.com> wrote:

> This stalled a bit, but the waters are still muddy...

So I've been messing with this a bit; wanted to do a proper patch posting
but I'm fried and mostly out of time for the moment.

The results I'm getting now can be seen at:

http://static.lwn.net/kerneldoc/

I've pulled in a few templates (including gpu.tmpl), converted them, and
built them into some reasonable-looking HTML, modulo a fair number of
glitches. There's lots of details to deal with, but the broad shape of it
is there. If you look, you'll see that things like cross-file
cross-references, a feature we've never had before, work nicely.

I can also get good EPUB and PDF output - except that rst2pdf is
currently crashing on me, which is a little discouraging. Man page
output will take more work.

What I have so far can be pulled from:

git://git.lwn.net/linux.git doc/sphinx

It's still based on using docproc because that was easiest (for me). The
kernel-doc part is Jani's asciidoc stuff, hacked up and made uglier. I'm
not sure that any of it is worth more than a demonstration of what can be
done; I'm not particularly proud of (or tied to) any of it. But it's a
start.

I've not looked at the table situation at all; soon.

jon

[Johannes Stezenbach]

Just a quick idea:
Perhaps one alternative would be to use Graphviz to render
the problematic tables, it supports a HTML-like syntax
and can be embedded in Spinx documents:

http://www.sphinx-doc.org/en/stable/ext/graphviz.html
http://www.graphviz.org/content/node-shapes#html
http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables


Johannes

[Mauro Carvalho Chehab]

Em Mon, 7 Mar 2016 00:29:08 +0100
Johannes Stezenbach <j...@linuxtv.org> escreveu:

> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> >
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> >
> > This is how this table should look like:
> > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> >
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> >
> > The RST files are on this git tree:
> > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/
>
> Yeah, seems it can't do merged cells in csv. Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.

Yes, the output is OK, but, as you said, working with tables in
Sphinx is hard, and using asciiart for the kind of tables we have
is not nice.

>
> Sphinx' latex writer can't handle nested tables, though.

Yeah, this is a big trouble that need to be solved if you're
willing to use Sphinx.

Btw, it crashes when trying to generate man pages:

Exception occurred:
File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", line 627, in depart_entry
self._active_table.append_cell(self.body[start:])
AttributeError: 'NoneType' object has no attribute 'append_cell'
The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you want to report the issue to the developers.

So, if we're willing to use sphinx, someone should either fix
it to produce latex nexted table and fix it to generate manpages,
or we'll need to stick with just html output.

> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.

At least here, rst2* scripts were unable to identify that the
index.rst had links to other rst documents.

In the specific case of rst2latex, I got several errors like:

index.rst:21: (ERROR/3) Unknown interpreted text role "ref".

> But I have doubts such a large
> table would render OK in pdf without using landscape orientation.

Yeah, in the past, when we had pdf enabled for DocBook (e. g. when
media development was using a separate mercurial tree), I guess
we had tags changing the text orientation on a few tables that
would otherwise won't diplay fine, but I can't remember the dirty
details anymore.

> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.
>
>
> Johannes

--
Thanks,
Mauro

[Mauro Carvalho Chehab]

Em Mon, 7 Mar 2016 09:48:26 +0100
Johannes Stezenbach <j...@linuxtv.org> escreveu:

That could work, but it is scary... Graphviz is great to generate
diagrams, but it really sucks when one wants to put a graph element
on a specific place, as it loves to reorder elements putting them on
unexpected places.

Btw,

I converted all docs from our uAPI docbook to rst using pandoc.
It was a brainless conversion, except for a few fixes.

The output is at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/

I added it on the top of my PoC tree at:
https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Besides tables, I noticed some other bad things that needs to be
corrected somehow:

1) Document divisions are not numbered. We need that. It should be
broken into:
- Document divisions - one per documented API:
- V4L2
- Remote Controllers
- DVB
- Media Controller

- Chapters
- Sessions

Everything should be numbered, as, when discussing API improvements,
it is usual the need of pinpoint to an specific chapter and section.

Tables and images should also be numbered, and we need a way to
use references for table/image numbers.

2) Images

Most images didn't popup. We have images on different file formats:
- SVG
- GIF
- PDF
- PNG

3) References

It could be a conversion issue, but there are lots of missing
references at the documentation.

4) We need to have some way to tell sphinx to not put some things
at the lateral ToC bar. For example, at the V4L2 "Changes" section,
we don't want to have one entry per version at the ToC bar.

Giving that, I suspect that we'll have huge headaches to address
if we use sphinx, as it seems too limited to handle complex
documents. We should try to use some other tool that would give
us better results.


Regards,
Mauro

[Russel Winder]

On Fri, 2016-03-04 at 09:46 +0200, Jani Nikula wrote:
> […]

> If we're talking about the same asciidoctor (http://asciidoctor.org/)
> it's written in ruby but you can apparently run it in JVM using
> JRuby. Calling it Java-based is misleading.

Indeed, I was somewhat imprecise. Thanks to the work mostly of Charles
Nutter, JRuby is invariably a faster platform for Ruby code than Ruby
is. So yes ASCIIDoctor is JVM-based via JRuby, not Java-based.

The real point here is that in a move from DocBook/XML as a
documentation source, ASCIIDoctor is an excellent choice.

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen <d...@opendevise.io> wrote:
> One of the key goals of the Asciidoctor project is to be able to directly
> produce a wide variety of outputs from the same source (without DocBook).
> We've added flexibility and best practices into the syntax and matured the
> converter mechanism to bridge this (sometimes very wide) gap.

I think our conclusion so far was that the native AsciiDoc (and
Asciidoctor) outputs fell short of our needs, forcing us to use the
DocBook pipeline. I, for one, was hoping we could eventually simplify
the toolchain. For example, there was no support for chunked, or split
to chapters, HTML, and the single page result was simply way too big.

> Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> acknowledge that (including the original creator).

Thanks for the input. We've touched the topic of AsciiDoc
vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
AsciiDoc. That actually makes choosing asciidoc harder, because
requiring another language environment complicates, not simplifies, the
toolchain. I'd really like to lower the bar for building the
documentation, for everyone, so much so that it becomes part of the
normal checks for patch inclusion.


BR,
Jani.


[1] http://mid.gmane.org/86pow31...@hiro.keithp.com

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 11:49:35 +0200
Jani Nikula <jani....@intel.com> escreveu:

As I failed to find a way to solve the issues with Sphinx/RST, I started
a PoC using Asciidoctor for the Media uAPI docbook conversion, in order
to see if this would work for us.

I'm not a big fan of using a language that I don't domain, like
Ruby (and the same applies to Python), but, as I said before,
provided that the toolchain works, can easily be installed on
Fedora and Debian, and provide the functionality I need, I'm
ok with that.

On my tests, Asciidoctor is really fast. It takes a fraction of the
time require to build from DocBook, with is a good thing.

Yet, I suspect that it doesn't have the strict checks that xmllint
have. For example, I didn't see any warning about a missing cross-ref.
We use those warnings to discover if something is added at the code,
but were not documented. Its error reports are also crappy, as it
doesn't tell where the problem is. For example:

$ asciidoctor -n -b docbook media_api.adoc
asciidoctor: WARNING: tables must have at least one body row

On a document with 33793 lines (this is the size of the uAPI doc
when converted to AsciiDoc format), the above warning doesn't help.

So, I'm actually planning to use AsciiDoc/xmllint to check for
documentation troubles, if possible.

Even with AsciiDoc/Asciidoctor, there are several features at the
media documents that aren't well supported. For example, we widely
use the Docbook's tags to generate a manpage-like description:

<refentry id="func-open">
<refmeta>
<refentrytitle>V4L2 open()</refentrytitle>
&manvol;
</refmeta>

<refnamediv>
<refname>v4l2-open</refname>
<refpurpose>Open a V4L2 device</refpurpose>
</refnamediv>

<refsynopsisdiv>
<funcsynopsis>
<funcsynopsisinfo>#include &lt;fcntl.h&gt;</funcsynopsisinfo>
<funcprototype>
<funcdef>int <function>open</function></funcdef>
<paramdef>const char *<parameter>device_name</parameter></paramdef>
<paramdef>int <parameter>flags</parameter></paramdef>
</funcprototype>
</funcsynopsis>
</refsynopsisdiv>

Asciidoctor doesn't produce anything like that, if the booktype is not
"manpage". Well, the media documentation is a multi-part book.

So, all those tags should be manually converted (actually, pandoc
made a mess with those tags - so manual work is required anyway,
no matter what markdown language we use). It will be a hard work,
but, at least, this is doable.

Also, it doesn't accept images with PDF format. We have a few PDF
images, although I we have them also in GIF. So, not a big issue here.

I also did lots of table conversions, to see if our complex tables
would fit. The answer is: Yes: on all cases I converted, it worked
fine.

There are two ways of doing that:
- Nested tables
- Merging cells

Some tables we use won't work fine with nested tables, as they have
cells merging the entire line of such tables.

Also, converting some tables to nested tables would be incredible hard,
as we have really big tables there that would need to add dozens of nested
tables inside, like this one:

https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4

Using merging cells works fine, tough, and it is not hard to use.
The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc
also accepts this) is:

.MFC 5.1 Control IDs
[width="100%",cols="7%,40%,13%,40%",options="header",]
|=======================================================================
|ID |Type 2+| Description

...

The "2+|" tells that the next content should be merged into two cells.

One of the good things is that we're not forced to use asciiart, with
would make it really hard to handle the tables.

For those wanting to see the results so far:

The html is at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

The testing tree is at:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/

PS.: pandoc did a really crap job on the conversion. To convert this
into something useful, we'll need to spend a lot of time, as it lost
most of the cross-references, as they were defined via DocBook macros.

It also dropped without even warning all nested tables. So, even for
a PoC, we'll need to spend some time to make it barely resembling
what we currently have on DocBook.

Thanks,
Mauro

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:09:40 -0700
Dan Allen <d...@opendevise.io> escreveu:

> Jani wrote:
>
> > there was no support for chunked, or split
> > to chapters, HTML, and the single page result was simply way too big.
> >
>

> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

The problem on pre-splitting the documents and process them in separate
is that this will break cross-references. At the media uAPI Docbook,
we use a lot of cross references.

Btw, we use a lot of includes. Currently, it has 187 separate files.

We even parse the header files looking for typedefs, structs, enums,
#defines and functions, in order to produce a document that will
cross-reference the documentation.

> Second, there is a custom converter in the works to split post-generate
> (which is really what we're talking about when we compare it to the DocBook
> toolchain).
>
> https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb
>
> It's just a prototype, but proves it is possible by design.

I didn't test it, but I saw some comments at the web that the part
that would handle cross-references between files is not ready.

> Personally, I don't like most chunked HTML approaches because they split
> arbitrarily. We are trying to find the right balance so that the output is
> actually sensible. There's still work to do, but there are options in the
> meantime.

Well, if it is capable of creating one chunk per include file, and do
cross-references between chunks, this would work for media UAPI book.

Yet, it would be good to have the multi-chunk extension packaged
on major distros, as I don't like the idea of installing it without
using my distro's package manager.

--
Thanks,
Mauro

[Mauro Carvalho Chehab]

Em Tue, 08 Mar 2016 05:13:13 -0700
Dan Allen <d...@opendevise.io> escreveu:

> On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab <
> mch...@osg.samsung.com> wrote:
>
> > pandoc did a really crap job on the conversion. To convert this
> > into something useful, we'll need to spend a lot of time, as it lost
> > most of the cross-references, as they were defined via DocBook macros.
> >
>

> I agree pandoc creates crappy AsciiDoc. We have a much better converter in
> the works called DocBookRx.
>
> https://github.com/opendevise/docbookrx
>
> It has converted several very serious DocBook documents and we're
> continuing to improve it. It's also a lot easier to hack than pandoc.

Didn't work:

$ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml
No visitor defined for <part>! Skipping.
No visitor defined for <part>! Skipping.
No visitor defined for <part>! Skipping.
No visitor defined for <part>! Skipping.
No visitor defined for <appendixinfo>! Skipping.


>
> -Dan
>
>


--
Thanks,
Mauro

[Jani Nikula]

On Tue, 08 Mar 2016, Dan Allen <d...@opendevise.io> wrote:

> That's not entirely true. First, you can pre-split at the source level
> using includes and generate output for each of the masters. That's what I
> tend to do and it works really well since these are logical split points.

I need to look into this again. Is there a specific option or directive
to produce split output for includes? When I tried this, the result was
just one big output file. (And indeed we'd need both. Some includes we
want embedded, some includes should produce separate outputs.)

>> That actually makes choosing asciidoc harder, because
>> requiring another language environment complicates, not simplifies, the
>> toolchain. I'd really like to lower the bar for building the
>> documentation, for everyone, so much so that it becomes part of the
>> normal checks for patch inclusion.
>

> Pardon my bluntness here, but I don't buy that argument. This is Linux.
> Installing software couldn't be simpler, and we're talking about an
> extremely well supported language (Ruby).

Granted, that part works for me. I'm not so sensitive to the
dependencies; others may disagree.

> I think it's a huge exaggeration to say that Asciidoctor is any harder to
> install than AsciiDoc Python. It's also a heck of a lot smaller in size
> since AsciiDoc Python pulls in hundreds of MB of LaTeX packages.

For me, the comparison is really between Sphinx and Asciidoctor, not so
much doc vs. doctor. The native output format and extension support in
Sphinx is appealing; I am not yet convinced we could manage with
Asciidoctor but without DocBook. The extension offering seems better in
Sphinx.

> Whatever you decide, I wish you all the best with your documentation
> efforts!

Thanks!

BR,
Jani.

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 10:39:22 -0300
Mauro Carvalho Chehab <mch...@osg.samsung.com> escreveu:

I tried to use docbookrx for the bits that were not properly converted,
like the manpage-like pages:

$ ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml
No visitor defined for <refentry>! Skipping.

Dan, if you want to take a look on what's going wrong here,
the XML I'm trying to convert is:

https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml

If this would work, it should be generating something like:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc

Pandoc failed to fully convert it, but at least it left all the texts,
with prevented rewriting it from scratch. This is the manual fix
I applied to it:
https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc

FYI, we have 133 xml files at the media uAPI doc with refmeta.

--
Thanks,
Mauro

[Jani Nikula]

On Wed, 09 Mar 2016, Dan Allen <d...@opendevise.io> wrote:

> On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula <jani....@intel.com> wrote:
>
>> I need to look into this again. Is there a specific option or directive
>> to produce split output for includes? When I tried this, the result was
>> just one big output file. (And indeed we'd need both. Some includes we
>> want embedded, some includes should produce separate outputs.)
>>
>

> Nope. What I'm saying is that you run Asciidoctor on each sub-master
> include file (an include that manages a part or chapter). That gives you
> your individual part/chapter files. Then you need to make an index page,
> probably by using the Asciidoctor API to itemize all the chapters as a list
> or something.

Bummer.

Getting the inter-document cross references right may become
tricky. We'll be generating plenty of snippets of lightweight markup
from source code documentation comments. At the time of processing, we
won't know where e.g. a specific function to be cross referenced is
documented, if at all. We can't require the documentation comment
writers to figure that out either; it's too burdensome, too ugly in the
code, and they'll bitrot quickly.

Cross referencing in the asciidoc proofs of concept have worked because
they've all done the processing as a single single unit, with
includes. These hacks have also ignored any broken links, and there have
been

> Yes, it does require some thinking about cross references. There is a lot
> more we can do out of the box, but all those references can be fixed with a
> little bit of post-processing in the meantime.

It seems to me Sphinx provides much better support regarding cross
references, out of the box, within documents and to external documents
(intersphinx), with target roles and domains, including validation and
not creating broken links in the output.

Looking at the current hacks we have for post-processing references, I'm
really not thrilled about the prospect of keeping or redoing that.

See how this works in Jon's Sphinx test [1]. At the time of generating
the markup from source comments, there is no idea if and where
gem_init_hw() and intel_guc_ucode_init() are documented. Indeed,
documentation for the former does not exist, but there's no broken link.

BR,
Jani.


[1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load

[Mauro Carvalho Chehab]

Em Tue, 8 Mar 2016 12:39:21 -0300

Mauro Carvalho Chehab <mch...@osg.samsung.com> escreveu:

> Pandoc failed to fully convert it, but at least it left all the texts,
> with prevented rewriting it from scratch. This is the manual fix
> I applied to it:
> https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc
>
> FYI, we have 133 xml files at the media uAPI doc with refmeta.

I used pandoc to convert from the html files and manually edited it.
I also fixed lots of other issues with the conversion.

I guess the conversion to asciidoc format is now in good shape,
at least to demonstrate that it is possible to use this format for the
media docbook. Still, there are lots of broken references.

The proof of concept html file is at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html

I also added the ascii doc files there, at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/

And I'm keeping the git tree, with helps to identify the work that was
needed to make it work:
https://git.linuxtv.org/mchehab/asciidoc-poc.git

In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce
an html file.

PROBLEMS
========

1)

I was not able to produce outputs on any other format.

For example, when trying to generate docbook45 output, it sems that
part of the trouble was due to pandoc conversion. It produces
links like:

link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^]

Which causes errors with DocBook parsers, like xmllint:

media_api.xml:32300: parser error : Opening and ending tag mismatch: superscript line 32300 and ulink
<ulink url="#id-1.4.11.43.5.11.2.7.2.6.2"><superscript>[a</ulink></superscript>]
^

I suspect that this is fixable. I may try to fix it later.

2) It seems that Asciidoctor doesn't allow annexes per document part.
It numberates them as chapters, instead of using A, B, C, ...

3) Even producing the html without troubles, it produces an error:
asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at least one section (e.g., chapter, appendix, etc.)

4) There are some things that got lost during the conversion, like
copyright notes and revision notes. This could be simply a problem
with pandoc conversion. Nothing serious, I guess, as we could insert
the lost data manually. Yet, it means that, to move from the PoC to
the Kernel, there are still lots of work to do.

I was unable do discover why, nor to suppress this error message.

Yet, from my side, if we're willing to get rid of DocBook, then
Asciidoctor seems to be the *only* alternative so far to parse the
complex media documents.

Regards,
Mauro

[Jani Nikula]

TL;DR? Skip to the last paragraph.

On Wed, 09 Mar 2016, Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:
> I guess the conversion to asciidoc format is now in good shape,
> at least to demonstrate that it is possible to use this format for the
> media docbook. Still, there are lots of broken references.

Getting references right with asciidoc is a big problem in the
kernel-doc side. As I wrote before, the proofs of concept only worked
because everything was processed as one big file (via includes). The
Asciidoctor inter-document references won't help, because we won't know
the target document name while processing kernel-doc.

Sphinx is massively better at handling cross references for
kernel-doc. We can use domains (C language) and roles (e.g. functions,
types, etc.) for the references, which provide kind of
namespaces. Sphinx warns for referencing non-existing targets, but
doesn't generate broken links in the result like Asciidoctor does.

For example, in the documentation for a function that has struct foo as
parameter or return type, a cross reference to struct foo is added
automagically, but only if documentation for struct foo actually
exists. In Asciidoctor, we would have to blindly generate the references
ourselves, and try to resolve broken links ourselves by somehow
post-processing the result.

> Yet, from my side, if we're willing to get rid of DocBook, then
> Asciidoctor seems to be the *only* alternative so far to parse the
> complex media documents.

I think you mean, "get rid of DocBook as source format", not altogether?
I'm yet to be convinved we could rely on Asciidoctor's native formats.

---

Mauro, I truly appreciate your efforts at evaluating both
alternatives. I also appreciate Dan's inputs on Asciidoctor.

Despite your evaluation that Asciidoctor is the only alternative for
media documents, it is my opinion that we should go with Sphinx.

It's an opinion, it's subjective, it's from my perspective, especially
from the kernel-doc POV, so please don't take it as a slap in the face
after all the work you've done. With that out of the way, here's why.

For starters, Jon's Sphinx proof-of-concept at
http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
usable. Cross references work, there are no broken links (I hacked a bit
more on kernel-doc and it gets even better). There's embedded search
(and if this gets exported to https://readthedocs.org/ the search is
even better). The API documentation is sensible and the headings aren't
mixed up with other headings. It's all there. It's what we've been
looking for.

The toolchain gets faster, easier to debug and simplified a lot with
DocBook out of the equation completely. Sphinx itself is stable, widely
available, and well documented. IMO there's sufficient native output
format support. There are plenty of really nice extensions
available. There's a possibility of doing kernel-doc as an extension in
the future (either by calling current kernel-doc from the extension or
by rewriting it).

Dan keeps bringing up the active community in Asciidoctor, and how
they're fixing things up as we speak... which is great, but Sphinx is
here now, packaged and shipping in distros ready to use. It seems that
of the two, an Asciidoctor based toolchain is currently more in need of
hacking and extending to meet our needs. Which brings us to the
implementation language, Python vs. Ruby.

I won't make the mistake of comparing the relative merits of the
languages, but I'll boldly claim the set of kernel developers who know
Python is likely larger than the set of kernel developers who know Ruby
[citation needed]. AFAICT there are no Ruby tools in the kernel tree,
but there is a bunch of Python. My own very limited and subjective
experience with other tools around the kernel is that Python is much
more popular than Ruby. So my claim here is that we're in a better
position to hack on Sphinx extensions ourselves than Asciidoctor.

My conclusion is that Sphinx covers the vast majority of the needs of
our documentation producers and consumers, in an amazing way, out of the
box, better than Asciidoctor.

Which brings us to the minority and the parts where Sphinx falls short,
media documentation in particular. It's complex documentation, with very
specific requirements on the output, especially that many things remain
exactly as they are now. It also feels like the target is more to have
standalone media documentation, and not so much to be aligned with and
be part of the rest of the kernel documentation.

I want to question the need to have all kernel documentation use tools
that meet the strict requirements of the outlier, when there's a better
alternative for the vast majority of the documentation. Especially when
Asciidoctor isn't a ready solution for media documentation either.

In summary, my proposal is to go with Sphinx, leave media docs as
DocBook for now, and see if and how they can be converted to
Sphinx/reStructuredText later on when we have everything else in
place. It's not the perfect outcome, but IMHO it's the best overall
choice.


BR,
Jani.

[Mauro Carvalho Chehab]

Em Thu, 10 Mar 2016 12:25:58 +0200
Jani Nikula <jani....@intel.com> escreveu:

> TL;DR? Skip to the last paragraph.
>
> On Wed, 09 Mar 2016, Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:
> > I guess the conversion to asciidoc format is now in good shape,
> > at least to demonstrate that it is possible to use this format for the
> > media docbook. Still, there are lots of broken references.
>
> Getting references right with asciidoc is a big problem in the
> kernel-doc side. As I wrote before, the proofs of concept only worked
> because everything was processed as one big file (via includes). The
> Asciidoctor inter-document references won't help, because we won't know
> the target document name while processing kernel-doc.

I was able to produce chunked htmls here with:

asciidoctor -b docbook45 media_api.adoc
xmlto -o html-dir html media_api.xml

The results are at:
https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/

But yeah, all references seem to be broken there. It could be due to some
conversion issue (I didn't actually tried to check what's wrong there),
but I think that there's something not ok with docbook45
output for multi-part documents (on both AsciiDoc and Asciidoctor).

> Sphinx is massively better at handling cross references for
> kernel-doc. We can use domains (C language) and roles (e.g. functions,
> types, etc.) for the references, which provide kind of
> namespaces. Sphinx warns for referencing non-existing targets, but
> doesn't generate broken links in the result like Asciidoctor does.
>
> For example, in the documentation for a function that has struct foo as
> parameter or return type, a cross reference to struct foo is added
> automagically, but only if documentation for struct foo actually
> exists. In Asciidoctor, we would have to blindly generate the references
> ourselves, and try to resolve broken links ourselves by somehow
> post-processing the result.
>
> > Yet, from my side, if we're willing to get rid of DocBook, then
> > Asciidoctor seems to be the *only* alternative so far to parse the
> > complex media documents.
>
> I think you mean, "get rid of DocBook as source format", not altogether?
> I'm yet to be convinved we could rely on Asciidoctor's native formats.

What I mean is that, right now, I see only two alternatives for the
media uAPI documentation:
1) keep using DocBook;
2) AsciiDoc/Asciidoctor.

Sphinx doesn't have what's needed to support the complexity of the
media books, specially since cell span seems to be possible only
by using asciiArt formats. Writing a big table using asciiArt is
something that is a *real pain*. Also, as tested, if the table is
too big, it fails to parse such asciiArt tables. So, while Sphinx
doesn't have a decent way to describe tables, we can't use it.

If it starts implementing it, then we can check if the other
features used by the media documentation are also supported.
Probably, multi-part books would be another pain with Sphinx.
We have actually 4 books inside a common body. A few chapters
(like book licensing, bibliography, error codes) are shared
by all 4 documents.

But, so far, I can't see any way to port media books without
lots of lot of work to develop new features at the Sphinx code.

> ---
>
> Mauro, I truly appreciate your efforts at evaluating both
> alternatives. I also appreciate Dan's inputs on Asciidoctor.
>
> Despite your evaluation that Asciidoctor is the only alternative for
> media documents, it is my opinion that we should go with Sphinx.
>
> It's an opinion, it's subjective, it's from my perspective, especially
> from the kernel-doc POV, so please don't take it as a slap in the face
> after all the work you've done. With that out of the way, here's why.
>
> For starters, Jon's Sphinx proof-of-concept at
> http://static.lwn.net/kerneldoc/ is pretty amazing. It's beautiful and
> usable. Cross references work, there are no broken links (I hacked a bit
> more on kernel-doc and it gets even better). There's embedded search
> (and if this gets exported to https://readthedocs.org/ the search is
> even better). The API documentation is sensible and the headings aren't
> mixed up with other headings. It's all there. It's what we've been
> looking for.
>
> The toolchain gets faster, easier to debug and simplified a lot with
> DocBook out of the equation completely. Sphinx itself is stable, widely
> available, and well documented. IMO there's sufficient native output
> format support. There are plenty of really nice extensions
> available. There's a possibility of doing kernel-doc as an extension in
> the future (either by calling current kernel-doc from the extension or
> by rewriting it).

Well, if we go to Sphinx for kernel-doc, that means that we'll need
2 different tools for the documentation:
- Sphinx for kernel-doc
- either DocBook or Asciidoctor/AsciiDoc for media.

IMHO, this is the worse scenario, as we'll keep depending on
DocBook plus requiring Sphinx, but it is up to Jon to decide.

> Dan keeps bringing up the active community in Asciidoctor, and how
> they're fixing things up as we speak... which is great, but Sphinx is
> here now, packaged and shipping in distros ready to use. It seems that
> of the two, an Asciidoctor based toolchain is currently more in need of
> hacking and extending to meet our needs. Which brings us to the
> implementation language, Python vs. Ruby.
>
> I won't make the mistake of comparing the relative merits of the
> languages, but I'll boldly claim the set of kernel developers who know
> Python is likely larger than the set of kernel developers who know Ruby
> [citation needed]. AFAICT there are no Ruby tools in the kernel tree,
> but there is a bunch of Python. My own very limited and subjective
> experience with other tools around the kernel is that Python is much
> more popular than Ruby. So my claim here is that we're in a better
> position to hack on Sphinx extensions ourselves than Asciidoctor.

Sorry, but I don't buy it. Python is, IMHO, a mess: each new version
is incompatible with the previous one, and requires the source to
change, in order to use a newer version than the one used to write
the code. So, when talking about Python, we're actually talking about
several different dialects that don't talk well to each other.

I don't know about Ruby. So far, I don't have anything against (or in
favor) of it. I bet most Kernel developers would actually prefer a
toolchain in C. If such tool doesn't exist, anything else seems
equally the same ;)

> My conclusion is that Sphinx covers the vast majority of the needs of
> our documentation producers and consumers, in an amazing way, out of the
> box, better than Asciidoctor.
>
> Which brings us to the minority and the parts where Sphinx falls short,
> media documentation in particular. It's complex documentation, with very
> specific requirements on the output, especially that many things remain
> exactly as they are now. It also feels like the target is more to have
> standalone media documentation, and not so much to be aligned with and
> be part of the rest of the kernel documentation.
>
> I want to question the need to have all kernel documentation use tools
> that meet the strict requirements of the outlier, when there's a better
> alternative for the vast majority of the documentation. Especially when
> Asciidoctor isn't a ready solution for media documentation either.
>
> In summary, my proposal is to go with Sphinx, leave media docs as
> DocBook for now, and see if and how they can be converted to
> Sphinx/reStructuredText later on when we have everything else in
> place. It's not the perfect outcome, but IMHO it's the best overall
> choice.

Well, this could be done. We don't have any good reason to move
the media docs out of DocBook. On the contrary, this means an extra
work. The only advantage is that it is a way simpler to write
documentation with a markup language, but converting from the PoC
to its integration at the Kernel tree still require lots of work,
specially due to the cross-refs "magic" scripts that we have under
Documentation/DocBook/media/Makefile.

As I said, the only big drawback is to keep depending on two
different tools for kernel-doc and for media documentation.

--
Thanks,
Mauro

[Markus Heiser]

Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc,
(mostly) you have more then one choice .. e.g. import csv tables ..
but this should be discussed by example ...

> If it starts implementing it, then we can check if the other
> features used by the media documentation are also supported.
> Probably, multi-part books would be another pain with Sphinx.
> We have actually 4 books inside a common body. A few chapters
> (like book licensing, bibliography, error codes) are shared
> by all 4 documents.
>
> But, so far, I can't see any way to port media books without
> lots of lot of work to develop new features at the Sphinx code.

may I can help you ...

>> The toolchain gets faster, easier to debug and simplified a lot with
>> DocBook out of the equation completely. Sphinx itself is stable, widely
>> available, and well documented. IMO there's sufficient native output
>> format support. There are plenty of really nice extensions
>> available. There's a possibility of doing kernel-doc as an extension in
>> the future (either by calling current kernel-doc from the extension or
>> by rewriting it).
>
> Well, if we go to Sphinx for kernel-doc, that means that we'll need
> 2 different tools for the documentation:
> - Sphinx for kernel-doc
> - either DocBook or Asciidoctor/AsciiDoc for media.
>
> IMHO, this is the worse scenario, as we'll keep depending on
> DocBook plus requiring Sphinx, but it is up to Jon to decide.
>

The migration of kernel-doc is a long term project, not a
one shot job. The scope of documents to migrate is not limited
to the files with DocBook markup in, most documents have not
a real markup.

Please take a look at my thoughts and efforts about migration.

* https://sphkerneldoc.readthedocs.org

* https://github.com/return42/sphkerneldoc.git

sphkerneldoc.git is a small project started this weekend, within
this project I show you, how migration could be done and
we can discuss concerns like "tables and cell-spans" by example.

Believe me, most concerns discussed in this thread are a leak of
knowledge. I'am working with sphinx-doc since 7 years, switched
over from DocBook (escaped from a 8 years lasting XML hell).
DocBook and sphinx-doc are complete different, so sphinx-doc
might feels odd in the first time, but if you have switched
like me, you will never go back again.

>> Dan keeps bringing up the active community in Asciidoctor, and how
>> they're fixing things up as we speak... which is great, but Sphinx is
>> here now, packaged and shipping in distros ready to use. It seems that
>> of the two, an Asciidoctor based toolchain is currently more in need of
>> hacking and extending to meet our needs. Which brings us to the
>> implementation language, Python vs. Ruby.
>>
>> I won't make the mistake of comparing the relative merits of the
>> languages, but I'll boldly claim the set of kernel developers who know
>> Python is likely larger than the set of kernel developers who know Ruby
>> [citation needed]. AFAICT there are no Ruby tools in the kernel tree,
>> but there is a bunch of Python. My own very limited and subjective
>> experience with other tools around the kernel is that Python is much
>> more popular than Ruby. So my claim here is that we're in a better
>> position to hack on Sphinx extensions ourselves than Asciidoctor.
>
> Sorry, but I don't buy it. Python is, IMHO, a mess: each new version
> is incompatible with the previous one, and requires the source to
> change, in order to use a newer version than the one used to write
> the code. So, when talking about Python, we're actually talking about
> several different dialects that don't talk well to each other.

Sorry, you are complete wrong ... I'am 15 years python programmer,
shipped out huge projects with my customers ... we never have seen
these problems ... sorry ...

> I don't know about Ruby. So far, I don't have anything against (or in
> favor) of it. I bet most Kernel developers would actually prefer a
> toolchain in C. If such tool doesn't exist, anything else seems
> equally the same ;)

Why we are talking about script languages? What needed is a
authoring system, which is as near as possible to the developers,
which are the authors.

Sphinx-Doc is a standard authoring-tool versioned, maintained
and extended by thousands of developers ...

>> My conclusion is that Sphinx covers the vast majority of the needs of
>> our documentation producers and consumers, in an amazing way, out of the
>> box, better than Asciidoctor.
>>
>> Which brings us to the minority and the parts where Sphinx falls short,
>> media documentation in particular. It's complex documentation, with very
>> specific requirements on the output, especially that many things remain
>> exactly as they are now. It also feels like the target is more to have
>> standalone media documentation, and not so much to be aligned with and
>> be part of the rest of the kernel documentation.
>>
>> I want to question the need to have all kernel documentation use tools
>> that meet the strict requirements of the outlier, when there's a better
>> alternative for the vast majority of the documentation. Especially when
>> Asciidoctor isn't a ready solution for media documentation either.
>>
>> In summary, my proposal is to go with Sphinx, leave media docs as
>> DocBook for now, and see if and how they can be converted to
>> Sphinx/reStructuredText later on when we have everything else in
>> place. It's not the perfect outcome, but IMHO it's the best overall
>> choice.
>
> Well, this could be done. We don't have any good reason to move
> the media docs out of DocBook.

Sorry but again wrong: you lost many of the authors which are
frustrated by a XML markup and you lost many developers to improve
the toolchain, frustrated by a complicated DocBook-XML XSLT
toolchain with SGML markup from the middle of the last epoch.

> On the contrary, this means an extra
> work. The only advantage is that it is a way simpler to write
> documentation with a markup language, but converting from the PoC
> to its integration at the Kernel tree still require lots of work,
> specially due to the cross-refs "magic" scripts that we have under
> Documentation/DocBook/media/Makefile.

Yes, you are right, migration is a process not a one shot
job, as I mentioned before. You are a great programmer, your
documentation is also great, this invest should be preserved.
So lets take a try. It would be a honor for me to show
you all theses steps by example on my repository (see above).

> As I said, the only big drawback is to keep depending on two
> different tools for kernel-doc and for media documentation.

-- Markus --

[Markus Heiser]

Hi kernel-doc authors,

motivated by this MT, I implemented a toolchain to migrate the kernel’s
DocBook XML documentation to reST markup.

It converts 99% of the docs well ... to gain an impression how
kernel-docs could benefit from, visit my sphkerneldoc project page
on github:

http://return42.github.io/sphkerneldoc/

The sources available at:

https://github.com/return42/sphkerneldoc

The work is underway, suggestions are welcome!

.. have a nice weekend ..

--M--

> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majo...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

[Hans Verkuil]

Hi Markus,

On 04/08/16 17:12, Markus Heiser wrote:
> Hi kernel-doc authors,
>
> motivated by this MT, I implemented a toolchain to migrate the kernel’s
> DocBook XML documentation to reST markup.
>
> It converts 99% of the docs well ... to gain an impression how
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
>
> http://return42.github.io/sphkerneldoc/
>
> The sources available at:
>
> https://github.com/return42/sphkerneldoc
>
> The work is underway, suggestions are welcome!

I have to admit that this looks pretty good :-)

My main remark based on my quick scan through the docs is that anything in
typewriter font seems to be shown as red text with a rectangle around it.
That's quite jarring for me and I think it should be just shown as normal
text, just using a non-proportional font, just like in the original.

I also noticed that the 'title' of tables ends with a '¶' character for
some reason.

See e.g. the struct v4l2_audioout table in
http://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/vidioc-g-audioout.html

Regards,

Hans

[Jonathan Corbet]

On Fri, 8 Apr 2016 17:12:27 +0200
Markus Heiser <markus...@darmarit.de> wrote:

> motivated by this MT, I implemented a toolchain to migrate the kernel’s
> DocBook XML documentation to reST markup.
>
> It converts 99% of the docs well ... to gain an impression how
> kernel-docs could benefit from, visit my sphkerneldoc project page
> on github:
>
> http://return42.github.io/sphkerneldoc/

So I've obviously been pretty quiet on this recently. Apologies...I've
been dealing with an extended death-in-the-family experience, and there is
still a fair amount of cleanup to be done.

Looking quickly at this work, it seems similar to the results I got. But
there's a lot of code there that came from somewhere? I'd put together a
fairly simple conversion using pandoc and a couple of short sed scripts;
is there a reason for a more complex solution?

Thanks for looking into this, anyway; I hope to be able to focus more on
it shortly.

jon

[Markus Heiser]

Hi Mauro, hi kernel-doc authors,

Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:

> Em Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser <markus...@darmarit.de> escreveu:

>
>> Hi kernel-doc authors,
>>
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>> DocBook XML documentation to reST markup.
>>
>> It converts 99% of the docs well ... to gain an impression how
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>>
>> http://return42.github.io/sphkerneldoc/
>>
>> The sources available at:
>>
>> https://github.com/return42/sphkerneldoc
>>
>> The work is underway, suggestions are welcome!
>>
>> .. have a nice weekend ..
>

> Hi Markus,
>
> Thanks for your work. I'm basically worried about the media docbook,
> with has some complexities that I was not able to figure out a way to
> convert it to reST.
>
> So, let me pinpoint some issues that I noticed there, on a quick
> look.
>
> The first thing I noticed is that the index doesn't match what's
> there, when generated from DocBook:
> https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html
>
> So, for instance, "Interfaces" is at the same level as "Input/Output".
>
> This sounds like an something went wrong when dealing with the title
> indentation levels during the conversion.
>

Yes, the hierarchical structure was broken by two causes. First was, that the
*chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) placed
all sections and sub(-sub)-sections at the same level (direct under chapter). This was
not only broken in the linux_tv book, in the other books also.

Thanks for pointing and sorry that I have overlooked this (I was so much focused on
on converting single elements to reST, that I not see the wood for the trees).

It is now fixed, may you give it a new try.


> I would also like to see the titles numbered in chapters (and, if
> possible, in sections and items) and to be properly indented at the
> index.

BTW a few words about differences between DockBook and reST (Sphinx).

With DocBook you write *books*, the protocol (the DocBook application) has
no facility to *chunk* and interconnect several documents. The external ENTITY
is a workaround on the SGML layer, not on XML nor on the DB-application layer.
Thats the reason, why so many XML-tools don't handle this entities and
many DocBook to (e.g.) reST tools are fail.

With **standard** reST it is nearly the same, except there is a "include"
directive on the application layer. But this directive is very simple,
comparable to the C preprocessor "#include" directive.

With the **superset** reST-markup of Sphinx-doc you get a the "toctree" directive,
which lets you control how a document-tree should be build.

http://www.sphinx-doc.org/en/stable/markup/toctree.html

@Mauro: you mentioned a docutils (rst2*) experience in your mail
http://marc.info/?l=linux-doc&m=145735316012094&w=2

Because the "toctree" directive -- and other directives
we use -- are a part of a superset of the **standard**
reST, the standard docutils (like rst2*) will not work.

OK, back to your requirements: within the toctree directive you can
set options like "maxdepth" and "numbered". It is a decision, how
deep TOCs should go and if they should be numbered. IMO, in a
HTML rendered page, with a proper navigation bar on the side, deep
TOCs in the running text have no pros, they only blow up the running
text and bring more scrolling with. In my sense numbering chapters
make only sense in books, not in HTML pages, where you have hyperlinks.

Just for demonstration, I added numbering in the linux-tv book:

https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3

May be, I will drop it later, because all reST sources are generated
by a make target and I always commit the whole reST tree. As I said,
it is a decision which might be made later, when the migration takes
places.

> Also, it seems that there's still a lot of work to do, as there are several
> tables that are missing conversion, like the table for "struct v4l2_pix_format":
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html
>
> and the big tables at:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
>

Yes, I marked them as TODO:

https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262

aspect *authoring tables* (see below)

> Also, some tables that are not so big like:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
>
> don't look nice, at least on my browser, as the "white" area is too small,
> and some cells seem to be broken in two, because of the color changes in the middle
> of the cell.

aspect *layout* (see below)

> On a side comment, I really think that editing those big tables using
> ASCIIart would be a real pain. It would be a way better to use some other
> table format at its source code.

It is the same to me (aspect *authoring tables*). IMO, it is a pain in
every markup not only in ASCII art. Markup tables by hand is good enough
for small tables and reST brings some variation to markup:

https://return42.github.io/sphkerneldoc/articles/table_concerns.html

There are also work-arounds like raw-html tables, but I'am not happy with
all this solutions. Authoring tables and the *layout aspect* are TOPICs I
will review later. I hope you see that I'am working on, but let me first
focus on other TOPICs like:

* man pages
* pdf
* kernel-doc reST output (done)
* etc ...

>
> Still on tables, you took an interesting approach with the tables with
> cell spans, like the control ones:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/controls.html
>

marginal note: due to redesigned chunks, link has changed:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/extended-controls.html

> Basically, you broke each control table (except for the first one) into a set
> of tables. Not sure if I liked it, but it is certainly a way of doing it.

Tables like: "Table 1.2. Codec Control IDs"

* http://www.linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html

are definition lists. Thats why I implemented handles for these tables
which made definition list from:

* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L432
* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L496

The "table" markup here is used to get and/or influence a specific layout.
I'am with you. As the time I wrote DocBooks, I also packed many things into
tables, but I had to realized, that not everything is a table. Please, be
openness for to change, I will be open for your requirements, except they
are IMHO wrong. Placing definition list into table is IMO wrong.

> There are also some "simple" tables at the media controller side that are
> missing conversion, like "struct media_v2_entity" at:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/media-ioc-g-topology.html

marginal note: due to redesigned chunks, link has changed:

* https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/media-ioc-g-topology.html?highlight=media_v2_entity#struct-media-v2-entity

> I would expect this to be easy to be converted, as there's not much
> weirdness on that.

Yes, no problem to convert them, but as I said, let me review all tables
later and focus first on the other TOPICs I mentioned above.

> There are also some things that didn't seem to be properly converted
> at:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/fdl-appendix.html
>

What's wrong with the license note? I can't remember tho old version,
but the new version should be OK?

* http://www.linuxtv.org/downloads/v4l-dvb-apis/fdl.html
* https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/fdl-appendix.html

> Do you see a way to fix the above issues with reST markup language, or
> this is something that can't be fixed?

In general: "Yes we can" ;-)

There will be points like tables we will have more discussions. With the
not yet covered points like pdf, man-pages and so force we will get more
questions to answer.

As long as we not try to reimplement DocBook in reST and with a focus on
productivity (which means, lets tend to pickup standards) we will find
suitable solutions.

I hope I could build confidence in reST. Please stay tuned and communicate
your requirements ...

--Markus--

> Thanks,
> Mauro

[Markus Heiser]

Hi Jonahtan,

Am 12.04.2016 um 17:46 schrieb Jonathan Corbet <cor...@lwn.net>:

> On Fri, 8 Apr 2016 17:12:27 +0200
> Markus Heiser <markus...@darmarit.de> wrote:
>
>> motivated by this MT, I implemented a toolchain to migrate the kernel’s
>> DocBook XML documentation to reST markup.
>>
>> It converts 99% of the docs well ... to gain an impression how
>> kernel-docs could benefit from, visit my sphkerneldoc project page
>> on github:
>>
>> http://return42.github.io/sphkerneldoc/
>
> So I've obviously been pretty quiet on this recently. Apologies...I've
> been dealing with an extended death-in-the-family experience, and there is
> still a fair amount of cleanup to be done.
>
> Looking quickly at this work, it seems similar to the results I got. But
> there's a lot of code there that came from somewhere?

From me? ... except the kernel-doc script which is a fork from your

git://git.lwn.net/linux.git doc/sphinx

> I'd put together a
> fairly simple conversion using pandoc and a couple of short sed scripts;
> is there a reason for a more complex solution?

It depends. If you have a simple DocBook with less various markup, maybe not.
May you want to read my remarks about migration tools and especially pandoc:

* https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc

A few more words about, what I have done:

I wrote a lib of XML filters which might be also usefully in other
migration projects (dbxml).

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py

It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because
I did not implemented a whole converter, I hacked around pandoc. Thats why
conversion is done in several steps:

1. copy xml file(s) to a cache space

2. substitude unsolved internal and external entities

3. filter all xml files

* run custom hooks on every node

* apply filters on every node and inject reST into the XML-tree where pandoc fails.
https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515

4. convert intermediary XML result with pandoc to json (needed by pandoc filters)

5. apply pandoc-filter and clean up the injected reST markup from step3

6. convert filtered json to reST

7. fix the produce reST with regular expression

... the last step is similar to your sed scripts.


And I wrote a commandline Interface to use this lib (see func db2rst):

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146

With this db2rst all kernel DB-XML books could be migrated, except the linux-tv
book, which has much more complexity. For this, there is a separated commandline
called media2rst

* https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107

The media2rst needs several special handlings, which is implemented in
hooks (the dbxml interface method)

* https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py

Summarize, why should one prefer this tools over pandoc + sed?

* Pandoc coverage is less on reading and writing, this is where
dbxml comes into play

- reading DocBook:
https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23

- writing reST has many bugs and leaks
(you fixed some of them with sed)

* Pandoc does not support external entities (linux-tv), covered by dbxml

* dbxml brings the ability to chunk one large XML book into small
reST chunks e.g. kernel-hacking book:

https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking

* dbxml lets you manipulate the XML source before you convert it to reST

this might helpfull e.g. if you have to convert single-column informal-tables
to lists or other things ... in short; dbxml and it's hooks are the key to hack
everything you need in a full automated DocBook-->reST migration workflow.


--Markus--

> Thanks for looking into this, anyway; I hope to be able to focus more on
> it shortly.
>
> jon

[Mauro Carvalho Chehab]

Em Mon, 18 Apr 2016 10:10:17 +0200
Markus Heiser <markus...@darmarIT.de> escreveu:

Thanks! It looks good now.

This is the uAPI spec DocBook, that we modify frequently, as we add
more features to the Kernel, and as we make sure that all drivers will
behave the same. So, from time to time, we need to clarify some topics
at the documentation. By having a numeration, it is easier for us to
discuss things like:
"1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly
described and requires some sort of clarification.

Ok, one could also refer to it via a hyperlink, but several Kernel
media maintainers prefer to generate a single big html file, as it
makes easier to locate everything it is needed on it.

So, with the item number, they can just seek for "1.2.10.14. " string
to find the item that is under discussion.

> > Also, it seems that there's still a lot of work to do, as there are several
> > tables that are missing conversion, like the table for "struct v4l2_pix_format":
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html
> >
> > and the big tables at:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> >
>
> Yes, I marked them as TODO:
>
> https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262
>
> aspect *authoring tables* (see below)
>
> > Also, some tables that are not so big like:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> >
> > don't look nice, at least on my browser, as the "white" area is too small,
> > and some cells seem to be broken in two, because of the color changes in the middle
> > of the cell.
>
> aspect *layout* (see below)

Ok.

> > On a side comment, I really think that editing those big tables using
> > ASCIIart would be a real pain. It would be a way better to use some other
> > table format at its source code.
>
> It is the same to me (aspect *authoring tables*). IMO, it is a pain in
> every markup not only in ASCII art. Markup tables by hand is good enough
> for small tables and reST brings some variation to markup:
>
> https://return42.github.io/sphkerneldoc/articles/table_concerns.html

Writing complex tables is painful, no matter what format it is use, but
doing ASCIIart would mean that, even a simple addition of a new column
may end by the need of adding additional whitespaces to every other column
in that table. That would be really painful for the developer, but not
only for him: All reviewers and maintainers will have troubles to identify
what are the relevant changes, and what changes are there just to make
reST happy.

Let me show you the big picture here: one thing that it is very common at
this book is that we frequently add new rows to the existing tables, in order
to add a new fourcc format, some new control, or some new colorspace format,
like this one:
https://git.linuxtv.org/media_tree.git/commit/?id=7146a9cfa499ac3bfaea18555d67afb04cac40c3

With has two new rows on one table:

--- a/Documentation/DocBook/media/v4l/pixfmt.xml
+++ b/Documentation/DocBook/media/v4l/pixfmt.xml
@@ -540,6 +540,10 @@ colorspaces except for BT.2020 which uses limited range R'G'B' quantization.</pa
<entry>See <xref linkend="col-bt2020" />.</entry>
</row>
<row>
+ <entry><constant>V4L2_COLORSPACE_DCI_P3</constant></entry>
+ <entry>See <xref linkend="col-dcip3" />.</entry>
+ </row>
+ <row>
<entry><constant>V4L2_COLORSPACE_SMPTE240M</constant></entry>
<entry>See <xref linkend="col-smpte-240m" />.</entry>
</row>
@@ -601,6 +605,10 @@ colorspaces except for BT.2020 which uses limited range R'G'B' quantization.</pa
<entry><constant>V4L2_XFER_FUNC_NONE</constant></entry>
<entry>Do not use a transfer function (i.e. use linear RGB values).</entry>
</row>
+ <row>
+ <entry><constant>V4L2_XFER_FUNC_DCI_P3</constant></entry>
+ <entry>Use the DCI-P3 transfer function.</entry>
+ </row>
</tbody>
</tgroup>
</table>

We don't want to see any changes on the patch touching other rows in
the table but the ones adding those new colorspace formats.

On the other hand, we seldom add new columns to the existing tables.
I can't remember any patch doing that on the last years. So, we
accept the extra pain to review patches adding/removing columns.

With that sense, the "List tables" format is also not good, as
one row addition would generate several hunks (one for each column
of the table), making harder to review the patch by just looking at
the diff.

The "CSV table" format seems to be the better one, as the produced
patches when a new one new row is added will be the best, except
that it currently reST CSV table format doesn't support cell span,
with is something that several of our tables use.

> There are also work-arounds like raw-html tables, but I'am not happy with
> all this solutions.

Yes, we might use raw-html tables, but that means that we'll never be
able to generate pdf. Well, for pdf generation, we would need to have a
tag at the markup language to tell to generate a "portrait" page for
some big tables. We used to have that before merging the V4L2 DocBook
upstream on some tables. Anyway, I would prefer to not add any
output format specific solution here.

So, I guess the best would be to extend reST language to do something
better. On the tests I did with AsciiDoc, I found the way it describes
tables good enough for the uses we have. Not sure if that approach
could be added or extended to reST/Sphinx.

There, a table with cell spans[1] would look like:

[width="100%",cols=",,,,",]
|===========================================
|start + 0: |Y'~00~ |Y'~01~ |Y'~02~ |Y'~03~
|start + 4: |Y'~10~ |Y'~11~ |Y'~12~ |Y'~13~
|start + 8: |Y'~20~ |Y'~21~ |Y'~22~ |Y'~23~
|start + 12: |Y'~30~ |Y'~31~ |Y'~32~ |Y'~33~
|start + 16: |Cr~00~ 3+|
|start + 17: |Cb~00~ 3+|
|===========================================

[1] https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/pixfmt-yuv410.adoc

On the above, the 3+| means that the cell will merge 3 columns.
Easy to write, easy to patch.

Also, it allows adding lines on big cells, so the above could be
written as:

[width="100%",cols=",,,,",]
|===========================================
|start + 0:
|Y'~00~ |Y'~01~ |Y'~02~ |Y'~03~
|start + 4: |Y'~10~ |Y'~11~ |Y'~12~ |Y'~13~
|start + 8: |Y'~20~ |Y'~21~ |Y'~22~ |Y'~23~
|start + 12:
|Y'~30~ |Y'~31~ |Y'~32~ |Y'~33~
|start + 16: |Cr~00~ 3+|
|start + 17: |Cb~00~ 3+|
|===========================================

And would produce the same output. That's actually a very good thing, as
we don't like to have lines with more than 80 columns withing the Kernel.

Another option would be to add some logic at the CSV to describe cell
spans at the existing reST markup format and add support on the CSV
format to break long lines (if not already supported).

A third approach would be to make sphinx to be able to fully understand
HTML tables and convert them to PDF and other formats, but I guess this
would be too much work (as, otherwise, someone would have done that
already).

> Authoring tables and the *layout aspect* are TOPICs I
> will review later. I hope you see that I'am working on, but let me first
> focus on other TOPICs like:
>
> * man pages
> * pdf
> * kernel-doc reST output (done)
> * etc ...

Yeah, I know that there are many things to be addressed.

Yet, for media maintainers, handling complex tables is a critical part for
the media books to be converted upstream. So, I'm a little anxious on that,
specially since I was unable to find a way to address with the current
markup language used on reST (except for raw-html, but, as I said before,
there are issues with that).

> >
> > Still on tables, you took an interesting approach with the tables with
> > cell spans, like the control ones:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/controls.html
> >
>
> marginal note: due to redesigned chunks, link has changed:
>
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/extended-controls.html
>

Noted.

> > Basically, you broke each control table (except for the first one) into a set
> > of tables. Not sure if I liked it, but it is certainly a way of doing it.
>
> Tables like: "Table 1.2. Codec Control IDs"
>
> * http://www.linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html
>
> are definition lists. Thats why I implemented handles for these tables
> which made definition list from:
>
> * https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L432
> * https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L496
>
> The "table" markup here is used to get and/or influence a specific layout.
> I'am with you. As the time I wrote DocBooks, I also packed many things into
> tables, but I had to realized, that not everything is a table. Please, be
> openness for to change, I will be open for your requirements, except they
> are IMHO wrong. Placing definition list into table is IMO wrong.

Yeah, I see your point. I guess your strategy would work. We'll likely
need to change a little bit the text for those "ex-tables" to make it
look better.

>
> > There are also some "simple" tables at the media controller side that are
> > missing conversion, like "struct media_v2_entity" at:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/media-ioc-g-topology.html
>
> marginal note: due to redesigned chunks, link has changed:
>
> * https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/media-ioc-g-topology.html?highlight=media_v2_entity#struct-media-v2-entity

Noted.

> > I would expect this to be easy to be converted, as there's not much
> > weirdness on that.
>
> Yes, no problem to convert them, but as I said, let me review all tables
> later and focus first on the other TOPICs I mentioned above.

Ok.

> > There are also some things that didn't seem to be properly converted
> > at:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/fdl-appendix.html
> >
>
> What's wrong with the license note? I can't remember tho old version,
> but the new version should be OK?
>
> * http://www.linuxtv.org/downloads/v4l-dvb-apis/fdl.html
> * https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/fdl-appendix.html

I'm actually referring to this:
https://www.linuxtv.org/downloads/v4l-dvb-apis/fdl-section4.html

The item numbering there on section 4 got weird on the conversion, as,
instead of:

". A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission. "

It became:

". A.

Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission."

Not a big issue. Just something for the TODO list.

> > Do you see a way to fix the above issues with reST markup language, or
> > this is something that can't be fixed?
>
> In general: "Yes we can" ;-)
>
> There will be points like tables we will have more discussions. With the
> not yet covered points like pdf, man-pages and so force we will get more
> questions to answer.
>
> As long as we not try to reimplement DocBook in reST and with a focus on
> productivity (which means, lets tend to pickup standards) we will find
> suitable solutions.
>
> I hope I could build confidence in reST. Please stay tuned and communicate
> your requirements ...

As I said, from my side, the big issue with reST is still tables format,
as I was not able yet to see a solution that would fit well for our
needs.

--
Thanks,
Mauro

[Grant Likely]

Hi Jon,

Thanks for digging into this. FWIW, here is my $0.02. I've been
working on restarting the devicetree specification, and after looking
at both reStructuredText and Asciidoc(tor) I thought I liked the
Asciidoc markup better, so chose that. I then proceeded to spend weeks
trying to get reasonable output from the toolchain. When I got fed up
and gave Sphinx a try, I was up and running with reasonable PDF and
HTML output in a day and a half.

Honestly, in the end I think we could make either tool do what is
needed of it. However, my impression after trying to do a document
that needs to have nice publishable output with both tools is that
Sphinx is easier to work with, simpler to extend, better supported. My
vote is firmly behind Sphinx/reStructuredText.

g.

[Daniel Vetter]

Hi all,

So sounds like moving ahead with rst/sphinx is the option that should
allow us to address everyone's concerns eventually? Of course the
first one won't have it all (media seems really tricky), but I'd like
to get something awesome in this area closer to mainline. I'm stalling
on typing more fancyful gpu docs right now (with tables, pictures and
stuff) since that would require a pile of needless work to redo when
we switch a few times more ;-)

And Jani also wants to get this in, but he doesn't really want to
spend more effort on a system that can't be merged.

So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

Cheers, Daniel

--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[Keith Packard]

Daniel Vetter <daniel...@ffwll.ch> writes:

> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer
> pov?

I think the right answer for today is to use sphinx to generate docs
From inline comments, to encourage outline docs to give it a try but to
allow doc writers to use whatever works for them.

That will leave the media docs using the existing system so that their
tables don't get wrecked, while providing guidance to new doc writers
and allowing inline docs to include markup.

--
-keith

[Markus Heiser]

Hi all, (hi Jonathan, please take note of my offer below)

Am 03.05.2016 um 16:31 schrieb Daniel Vetter <daniel...@ffwll.ch>:

> Hi all,
>
> So sounds like moving ahead with rst/sphinx is the option that should
> allow us to address everyone's concerns eventually? Of course the

> first one won't have it all (media seems really tricky), ...

BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ...

Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:

> With that sense, the "List tables" format is also not good, as
> one row addition would generate several hunks (one for each column
> of the table), making harder to review the patch by just looking at
> the diff.

For this, I wrote the "flat-table" reST-directive, which adds
missing cells automatically:

doc: http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
source: https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

I used "flat-table" to migrate all DocBook-XML documents to reST. With this
directive, I also managed to migrate the complete media book (no more TODOs)
incl. the large tables like them from subdev-formats:

https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html

(Rendering large tables is a general discussion which should not take place in this MT)

> but I'd like
> to get something awesome in this area closer to mainline. I'm stalling
> on typing more fancyful gpu docs right now (with tables, pictures and
> stuff) since that would require a pile of needless work to redo when
> we switch a few times more ;-)

Are your "fancyful gpu" written from scratch, or will it be an rewrite
of the Documentation/DocBook/gpu.tmpl?

If it is the last; as starting point you can use a copy of:

https://github.com/return42/sphkerneldoc/tree/master/doc/books/gpu

but I think this will not by very helpful, as long as you miss
a similar ".tmpl" workflow for reST documents.

I'am working on a reST directive (named: "kernel-doc") to provide a
similar ".tmpl" workflow within plain reST. The first step towards
is done with (my) modified kernel-doc script ...

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

which produce reST from source code comments. E.g. this content is
generated with it.

* http://return42.github.io/sphkerneldoc/linux_src_doc/index.html

> And Jani also wants to get this in, but he doesn't really want to
> spend more effort on a system that can't be merged.
>
> So sphinx/rst y/n? Jon, is that ok with you from the doc maintainer pov?

My recommendation is to start with reST-markup if you are
writing new docs from scratch. Workflows like the .tmpl one
can be added to the kernel build process step by step and
there is no need to drop the DocBook-XML process. Every author
should be free to decide by himself, when it is time to
migrate his DocBook sources into a reST building process.

I'am new to the kernel, my aim is to support the doc maintainer
(contact me), e.g. to migrate existing DocBooks to reST,
elaborate (build) procedures for improved reST support
within the kernel sources and to give recommendation to
infrastructures.

@Jonathan: I know, you have no time right now. If it is OK
for you, I elaborate all these reST stuff within my POC
at github (this will take some time). Later, if you have
time, we can merge the made experience and tools to the
kernel build process .. is that okay for you?

-- Markus --

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> but I think this will not by very helpful, as long as you miss
> a similar ".tmpl" workflow for reST documents.
>
> I'am working on a reST directive (named: "kernel-doc") to provide a
> similar ".tmpl" workflow within plain reST. The first step towards
> is done with (my) modified kernel-doc script ...
>
> * https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>
> which produce reST from source code comments. E.g. this content is
> generated with it.

What do you mean by ".tmpl workflow"?

I'd be *very* hesitant about adding the kind of things you do in
reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
comments must be as simple as possible with basically pass-through of
the comment blocks to sphinx. Specifically, do not attempt to detect and
parse elements like lists in kernel-doc.

[Markus Heiser]

Hi Jani,

Am 04.05.2016 um 11:58 schrieb Jani Nikula <jani....@intel.com>:

> On Wed, 04 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
>> but I think this will not by very helpful, as long as you miss
>> a similar ".tmpl" workflow for reST documents.
>>
>> I'am working on a reST directive (named: "kernel-doc") to provide a
>> similar ".tmpl" workflow within plain reST. The first step towards
>> is done with (my) modified kernel-doc script ...
>>
>> * https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736
>>
>> which produce reST from source code comments. E.g. this content is
>> generated with it.
>
> What do you mean by ".tmpl workflow"?

Sorry for bad naming, I addressed the DOCPROC build process which builds
the .xml files from .tmpl files ...

<SNIP> ----
###
# The build process is as follows (targets):
# (xmldocs) [by docproc]
# file.tmpl --> file.xml +--> file.ps (psdocs) [by db2ps or xmlto]
# +--> file.pdf (pdfdocs) [by db2pdf or xmlto]
# +--> DIR=file (htmldocs) [by xmlto]
# +--> man/ (mandocs) [by xmlto]
....
###
# DOCPROC is used for two purposes:
# 1) To generate a dependency list for a .tmpl file
# 2) To preprocess a .tmpl file and call kernel-doc with
# appropriate parameters.
# The following rules are used to generate the .xml documentation
# required to generate the final targets. (ps, pdf, html).
<SNAP> -----


The DOCPROC markup directives are described in the kernel-doc-nano-HOWTO:

https://github.com/torvalds/linux/blob/master/Documentation/kernel-doc-nano-HOWTO.txt#L332

My aim is to implement a reST-directive which is similar. E.g:

<XML-SNIP> -----
<sect2>
<title>Device Instance and Driver Handling</title>
!Pdrivers/gpu/drm/drm_drv.c driver instance overview
!Edrivers/gpu/drm/drm_drv.c
</sect2>
<sect2>
<XML-SNAP> -----

In reST the directive might look like:

<reST-SNIP> -----
Device Instance and Driver Handling
===================================

.. kernel-doc:: drivers/gpu/drm/drm_drv.c
:doc: driver instance overview
:exported:

<reST-SNAP> -----

> I'd be *very* hesitant about adding the kind of things you do in
> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc
> comments must be as simple as possible with basically pass-through of
> the comment blocks to sphinx.

Right, but you forgot, that the current markup in the source code comments
is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all
these source code comments to reST markup ;-)

So there is a need to differentiate between *vintage* kernel-doc markup
and reST markup.

My suggestion is to add a tag to the comments, here a short example
what this might look like.

<vintage-comment-SNIP> --------------
/**
* drm_minor_release - Release DRM minor
* @minor: Pointer to DRM minor object
*
* Release a minor that was previously acquired via drm_minor_acquire().
*/
<vintage-comment-SNAP> --------------

in short: the vintage style does not need any change and
comments with reST markup has a tag ":reST:" in the first
line(s) ...

<reST-comment-SNIP> --------------
/**
* :reST:
*
* .. c:function:: void drm_minor_release(struct drm_minor *minor)
*
* Release DRM minor
*
* :param struct drm_minor \*minor: Pointer to DRM minor object
*
*/
<reST-comment-SNAP> --------------

Comments with the ":reST:" tag could be exported and pass-through

to sphinx.

> Specifically, do not attempt to detect and
> parse elements like lists in kernel-doc.

If your markup in the comments is plain reST, no need to parse, but there
are markups in the (vintage) comments which made use of individual
text-markups, e.g. the markup of lists or ASCII-art.

This individual text-markups has not been converted/rendered in the docbook
output, but I wanted to convert this individuals to reST, to render them in
Sphinx.

E.g. compare the member & description section of struct drm-display-mode ...

DocBook:

* https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html

kernel-doc reST with the additional reformat_block_rst function:

* http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode


-- Markus--

> --
> Jani Nikula, Intel Open Source Technology Center

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
>> What do you mean by ".tmpl workflow"?
>
> Sorry for bad naming, I addressed the DOCPROC build process which builds
> the .xml files from .tmpl files ...

Yeah, I know more about this part than I care. I was just wondering what
you refer to with that in the sphinx context.

> In reST the directive might look like:
>
> <reST-SNIP> -----
> Device Instance and Driver Handling
> ===================================
>
> .. kernel-doc:: drivers/gpu/drm/drm_drv.c
> :doc: driver instance overview
> :exported:
>
> <reST-SNAP> -----

Yes, I think something like this, parsed by sphinx directly (via some
extension perhaps), should be the end goal. I am not sure if it should
be the immediate first goal though or whether we should reuse the
existing docproc for now.

Disagreed on most of the above.

Maybe I was unclear in my previous mail, and I've probably worked on
this so much previously that I thought it was clear how we should handle
lightweight markup in kernel-doc comments.

Kernel-doc the tool should continue to parse kernel-doc comments at the
high level like they currently are, according to the kernel-doc
howto. The first heading line, the parameter/member lines with @param:,
and references within the text with @param, function(), &struct
structures, etc. For those, kernel-doc should produce appropriate rst
elements.

Beyond that, kernel-doc should *not* try to make guesses about the
formatting of the documentation comments. It should just pass the rest
of the formatting through. If the ad-hoc lists etc. in the current
comments don't turn into proper rst lists, then so be it. If it bugs
people, the comments will gradually be converted to proper rst. The
worst we could do is promote a sloppy style that kernel-doc fixes for
you, possibly "fixing" things you'd want to pass through to sphinx.

What you have in <reST-comment-SNAP> example above will not fly. The
documentation comment style must remain as is defined in the kernel-doc
how, i.e. <vintage-comment-SNAP> example. It is imperative that the
kernel-doc comments remain as readable as they currently are in the
source code.

>> Specifically, do not attempt to detect and
>> parse elements like lists in kernel-doc.
>
> If your markup in the comments is plain reST, no need to parse, but there
> are markups in the (vintage) comments which made use of individual
> text-markups, e.g. the markup of lists or ASCII-art.
>
> This individual text-markups has not been converted/rendered in the docbook
> output, but I wanted to convert this individuals to reST, to render them in
> Sphinx.
>
> E.g. compare the member & description section of struct drm-display-mode ...
>
> DocBook:
>
> * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
>
> kernel-doc reST with the additional reformat_block_rst function:
>
> * http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

Overall I think we should promote fixing those in the kernel-doc
comments. Trying to guesswork in kernel-doc tool will leave the source
littered with bad examples that get proliferated all around. Instead of
gradually fixing things, we'll end up gradually messing things up even
more.

For those specific examples, IIRC the last time I played with that,
simply leaving the prefix whitespace in place went a long way (just eat
" * " from the lines). The asciiart just worked.

BR,
Jani.

[Daniel Vetter]

I think we need to unconfuse what's current standardize kerneldoc markup.
There's three bits:
- The header with the one-liner and parameters, i.e.

/**
* drm_minor_release - Release DRM minor
* @minor: Pointer to DRM minor object

There's thousands of those, and we cant realistically change them. This
means we need to teach kernel-doc to parse those and convert them to
rest/shpinx layout. Any approach that expects a conversion of all the
existing comments over to sphinx/rst is imo unworkable.

- References for function(), &structures, #constants and @parameters.
Again for the same reasons of being here already, we can't change those
at all. On top of that these markers are also used to generate
hyperlinks in the final docs. The kernel-doc script therefor not only
needs to generate the right markup, but also correct links. Without
generating links to functions/structures outside of a given document.

- kernel-doc also does paragraph splitting and minimalistic headers like

* Returns:
*
* This will be under a separate Returns: heading in the text.

afaict at least the paragraph splitting would be identical between all
proposed markup languages.

Now what all current proposals have done is simply allow pass-through of
asciidoc or rst markup in those paragraphs, so that the final processor
could make stuff pretty.

This is already used widely in kerneldoc included by gpu.tmpl, and
currently it's asciidoc. Those lists and asciiarts though are _not_
standard kerneldoc at all. But imo any doc toolchain improvement should
integrate along those lines.

For reference, this is what it's supposed to look like with the asciidoc
support enabled:

https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html

Fyi those hacked-up patches to make this happen are available in

https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc

Note that this isn't upstream, but we did start using this support all
over in gpu documentation simply because we really, really need it. And
I'm willing to throw all the comments into a converter to end up with
whatever markup we decide upon.

But what will not work is to throw the existing kernel-doc standard into
the shredder. So all the stuff with @, (), & and # must keep working.

> E.g. compare the member & description section of struct drm-display-mode ...
>
> DocBook:
>
> * https://www.kernel.org/doc/htmldocs/gpu/API-struct-drm-display-mode.html
>
> kernel-doc reST with the additional reformat_block_rst function:
>
> * http://return42.github.io/sphkerneldoc/linux_src_doc/include/drm/drm_modes_h.html#struct-drm-display-mode

So we have this already. The two things I thought this entire discussion
was about:

- also throw out the docbook .xml and replace it with something else. That
was jani's prototype on top of asciidoc, and it sounds like we could do
something similar with rst/sphinx. Everyone pretty much agrees that
this would be a second step, and the first step would be to simply
integrate more advanced markup into the existing toolchain.

- which flavour of markup has the best support for more advanced use-cases
like those in the media docbook. It sounds like sphinx won that
competition.

I'd really like to converge on the markup question, so that we can start
using all the cool stuff with impunity in gpu documentations.
-Daniel

--
Daniel Vetter
Software Engineer, Intel Corporation

http://blog.ffwll.ch

[Daniel Vetter]

On Wed, May 4, 2016 at 3:43 PM, Daniel Vetter <dan...@ffwll.ch> wrote:
> I'd really like to converge on the markup question, so that we can start
> using all the cool stuff with impunity in gpu documentations.

Aside: If we decide this now I could send in a pull request for the
rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
markdown/asciidoc code I still have). That would be really awesome ...

Jon?

Thanks, Daniel

--
Daniel Vetter
Software Engineer, Intel Corporation

[Jonathan Corbet]

On Wed, 4 May 2016 16:18:27 +0200
Daniel Vetter <daniel...@ffwll.ch> wrote:

> > I'd really like to converge on the markup question, so that we can start
> > using all the cool stuff with impunity in gpu documentations.
>
> Aside: If we decide this now I could send in a pull request for the
> rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> markdown/asciidoc code I still have). That would be really awesome ...

Sorry for my relative absence...I'm still busy dealing with bureaucracy
an ocean away from home. I hope to begin emerging from this mess in the
near future.

So ... there's the code you have, the work I (+Jani) did, and the work
Markus has done. Which would you have me push into 4.7?

The sphinx/rst approach does seem, to me, to be the right one, with the
existing DocBook structure remaining in place for those who want/need
it. I'm inclined toward my stuff as a base to work with, obviously :) But
it's hackish at best and needs a lot of cleaning up. It's a proof of
concept, but it's hardly finished (one might say it's barely begun...)

In the end, I guess, I feel that anything we might try to push for 4.7 is
going to look rushed and not ready, and Linus might react accordingly.
I'd be more comfortable aiming for 4.8. I *will* have more time to focus
on things in that time frame... I suspect you're pretty well fed up with
this stuff being pushed back, and rightly so. All I can do is apologize.

That said, if you do think there's something out there that is good
enough to consider pushing in a week or two, do tell and we can all take
a look.

Thanks,

jon

[Jonathan Corbet]

On Wed, 04 May 2016 16:41:50 +0300
Jani Nikula <jani....@intel.com> wrote:

> On Wed, 04 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> > In reST the directive might look like:
> >
> > <reST-SNIP> -----
> > Device Instance and Driver Handling
> > ===================================
> >
> > .. kernel-doc:: drivers/gpu/drm/drm_drv.c
> > :doc: driver instance overview
> > :exported:
> >
> > <reST-SNAP> -----
>
> Yes, I think something like this, parsed by sphinx directly (via some
> extension perhaps), should be the end goal. I am not sure if it should
> be the immediate first goal though or whether we should reuse the
> existing docproc for now.

I think all of this makes sense. It would be really nice to have the
directives in the native sphinx language like that. I *don't* think we
need to aim for that at the outset; the docproc approach works until we can
properly get rid of it. What would be *really* nice would be to get
support for the kernel-doc directive into the sphinx upstream.

> > <reST-comment-SNAP> --------------
> >
> > Comments with the ":reST:" tag could be exported and pass-through
> > to sphinx.
>
> Disagreed on most of the above.

Agreed with the disagreement here. We can't be adding ":reST:" tags to
comments; I anticipate a wee bit of pushback if we try. It needs to work
with the comments as they are now. It seems that should be possible.

Thanks,

jon

[Daniel Vetter]

Well I'd just have taken the asciidoc hacks I have currently in my
topic/kerneldoc branch, converted to sphinx and looked how it fares. It
should be fairly minimal, and I think the first step we want to do for the
long-term plan. I hope I can ready something, and then we can look whether
it's rushed for 4.7 or not.

Thanks, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation

http://blog.ffwll.ch

[Jani Nikula]

On Wed, 04 May 2016, Jonathan Corbet <cor...@lwn.net> wrote:
> The sphinx/rst approach does seem, to me, to be the right one, with the
> existing DocBook structure remaining in place for those who want/need
> it. I'm inclined toward my stuff as a base to work with, obviously :) But
> it's hackish at best and needs a lot of cleaning up. It's a proof of
> concept, but it's hardly finished (one might say it's barely begun...)

Thanks. I'll start looking at how to make it less hackish and more
upstreamable.

BR,
Jani.

[Markus Heiser]

OK, forget my <reST-comment-SNAP> example, I don't really want to promote
this way ... I agree, it is better to stay with standardize kernel-doc markup
and provide a "pass through" for the section-content (even if it is mixing
markups).

Am 04.05.2016 um 15:41 schrieb Jani Nikula <jani....@intel.com>:

> Kernel-doc the tool should continue to parse kernel-doc comments at the
> high level like they currently are, according to the kernel-doc
> howto.

The requirement was unclear for me, thanks to Daniel and
Jani for clarifying this.

Am 04.05.2016 um 15:43 schrieb Daniel Vetter <dan...@ffwll.ch>:

> This is already used widely in kerneldoc included by gpu.tmpl, and
> currently it's asciidoc. Those lists and asciiarts though are _not_
> standard kerneldoc at all. But imo any doc toolchain improvement should
> integrate along those lines.
>
> For reference, this is what it's supposed to look like with the asciidoc
> support enabled:
>
> https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html
>
> Fyi those hacked-up patches to make this happen are available in
>
> https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc
>

Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
particular with your "MARKDOWNREADY := gpu.xml" process.

As I understood, you convert markdown to docbook within the kernel-doc
script using pandoc's executable? ... I don't face this topic. With my
modification of kernel-doc I produced pure reST markup from standardize
kernel-doc markup, no intermediate steps or tools required.

Am 04.05.2016 um 15:41 schrieb Jani Nikula <jani....@intel.com>:

> Overall I think we should promote fixing those in the kernel-doc
> comments. Trying to guesswork in kernel-doc tool will leave the source
> littered with bad examples that get proliferated all around. Instead of
> gradually fixing things, we'll end up gradually messing things up even
> more.

I agree with you, lets drop the reformat_block_rst from my kernel-doc script:

* https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736

and we should have a plain "pass through".

Am 04.05.2016 um 17:09 schrieb Jonathan Corbet <cor...@lwn.net>:

> I think all of this makes sense. It would be really nice to have the
> directives in the native sphinx language like that. I *don't* think we
> need to aim for that at the outset; the docproc approach works until we can
> properly get rid of it. What would be *really* nice would be to get
> support for the kernel-doc directive into the sphinx upstream.

No need for kernel-doc directive in sphinx upstream, later it will be
an extension which could be installed by a simple command like
"pip install kernel-doc-extensions" or similar.

I develop these required extension (and more) within my proof of concept
on github ... this takes time ... if I finished all my tests and all is
well, I will build the *kernel-doc-extensions* package and deploy it
on https://pypi.python.org/pypi from where everyone could install this
with "pip".

At this time I see only one change/merge to the linux upstream, this is my
modification of the kernel-doc script to get a proper reST output.

But my recommendation is not to merge anything in a hurry.

--Markus--

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 11:34:08 +0200
Markus Heiser <markus...@darmarit.de> escreveu:

> Hi all, (hi Jonathan, please take note of my offer below)
>
> Am 03.05.2016 um 16:31 schrieb Daniel Vetter <daniel...@ffwll.ch>:
>
> > Hi all,
> >
> > So sounds like moving ahead with rst/sphinx is the option that should
> > allow us to address everyone's concerns eventually? Of course the
> > first one won't have it all (media seems really tricky), ...
>
> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ...
>
> Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:
>
> > With that sense, the "List tables" format is also not good, as
> > one row addition would generate several hunks (one for each column
> > of the table), making harder to review the patch by just looking at
> > the diff.
>
> For this, I wrote the "flat-table" reST-directive, which adds
> missing cells automatically:
>
> doc: http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
> source: https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py

Yeah, this should address the lack of a proper way to markup cell/row
spans, providing the additional bits for the tables we have at media.

Yet, there are some issues with table conversions. See below.

>
> I used "flat-table" to migrate all DocBook-XML documents to reST. With this
> directive, I also managed to migrate the complete media book (no more TODOs)
> incl. the large tables like them from subdev-formats:
>
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
>
> (Rendering large tables is a general discussion which should not take place in this MT)

Some tables, like the one here:
https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html

are truncated (tested with Mozilla and Chrome), and part of the information is
lost due to that.

Regards,
Mauro

[Jani Nikula]

On Wed, 04 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in
> particular with your "MARKDOWNREADY := gpu.xml" process.
>
> As I understood, you convert markdown to docbook within the kernel-doc
> script using pandoc's executable? ... I don't face this topic. With my
> modification of kernel-doc I produced pure reST markup from standardize
> kernel-doc markup, no intermediate steps or tools required.

That pandoc thing is a dead end. Forget about it. I think we've all
pretty much agreed we should have kernel-doc produce the lightweight
markup directly since [1].

[1] http://mid.gmane.org/1453106477-21359-1-git...@intel.com

> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet <cor...@lwn.net>:
>
>> I think all of this makes sense. It would be really nice to have the
>> directives in the native sphinx language like that. I *don't* think we
>> need to aim for that at the outset; the docproc approach works until we can
>> properly get rid of it. What would be *really* nice would be to get
>> support for the kernel-doc directive into the sphinx upstream.
>
> No need for kernel-doc directive in sphinx upstream, later it will be
> an extension which could be installed by a simple command like
> "pip install kernel-doc-extensions" or similar.
>
> I develop these required extension (and more) within my proof of concept
> on github ... this takes time ... if I finished all my tests and all is
> well, I will build the *kernel-doc-extensions* package and deploy it
> on https://pypi.python.org/pypi from where everyone could install this
> with "pip".

I think we should go for vanilla sphinx at first, to make the setup step
as easy as possible for everyone. Even if it means still doing that ugly
docproc step to call kernel-doc. We can improve from there, and I
definitely appreciate your work on making this work with sphinx
extensions.

That said, how would it work to include the kernel-doc extension in the
kernel source tree? Having things just work if sphinx is installed is
preferred over requiring installation of something extra from pypi. (I
know this may sound backwards for a lot of projects, but for kernel I'm
pretty sure this is how it should be done.)

BR,
Jani.

[Daniel Vetter]

Ok, discussed this a bit more with Jani on IRC and he really doesn't
like the old design of that branch (it calls the converter for every
kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully
something for 4.8.
-Daniel

--
Daniel Vetter
Software Engineer, Intel Corporation

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 08:57:13 -0600
Jonathan Corbet <cor...@lwn.net> escreveu:

> On Wed, 4 May 2016 16:18:27 +0200
> Daniel Vetter <daniel...@ffwll.ch> wrote:
>
> > > I'd really like to converge on the markup question, so that we can start
> > > using all the cool stuff with impunity in gpu documentations.
> >
> > Aside: If we decide this now I could send in a pull request for the
> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal
> > markdown/asciidoc code I still have). That would be really awesome ...
>
> Sorry for my relative absence...I'm still busy dealing with bureaucracy
> an ocean away from home. I hope to begin emerging from this mess in the
> near future.
>
> So ... there's the code you have, the work I (+Jani) did, and the work
> Markus has done. Which would you have me push into 4.7?

IMHO, Markus approach is the one that is providing a better result,
as it added support for missing features that we require for the
media DocBook.

Still, it seems premature to merge it for 4.7.

Markus is getting real progress on converting the media docs to work
with Sphynx, but there are still troubles when the table is big, as,
currently, it is truncating everything that passes the right margin
on some tables.

So, IMHO, the next action item would be to be sure that big tables
will not be truncated. Assuming that he can fix that in time, we
can merge it for 4.8, and then start porting the documentation to
use it.

Btw, converting the DocBook/media Makefile will require an extra
step, as part of the documentation is generated via scripts (some
C file examples, and include/uapi headers). Those scripts also
warrant that (almost) every API at include/uapi is synchronized
with the DocBook. I use this on my patch review process, in order
to reject patches that don't add the proper documentation updates.

Thanks,
Mauro

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 19:13:21 +0300
Jani Nikula <jani....@intel.com> escreveu:

Vanilla Sphinx doesn't work, as reST markup language is too limited
to support the docs we have. So, we should either push the needed
extensions to Sphinx reST or find a way to put it at Kernel tree
without causing too much pain for the developers, e. g. as something
that just doing "make htmldoc" would automatically use such extensions
without needing to actually install the extensions.

> Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

I disagree: We should not cause documentation regressions by
producing incomplete documentation and losing tables because of
such conversion.

The quality of the documentation after the change should be *at least*
equal to the one we current produce, never worse.

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Yeah, using pypi seems an additional undesired step. Aren't there
any way to make python to use an additional extension at the
Kernel tree without needing to install it?

Thanks,
Mauro

[Jonathan Corbet]

On Wed, 4 May 2016 13:50:35 -0300

Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:

> Em Wed, 4 May 2016 19:13:21 +0300
> Jani Nikula <jani....@intel.com> escreveu:

> > I think we should go for vanilla sphinx at first, to make the setup step
> > as easy as possible for everyone.
>
> Vanilla Sphinx doesn't work, as reST markup language is too limited
> to support the docs we have. So, we should either push the needed
> extensions to Sphinx reST or find a way to put it at Kernel tree
> without causing too much pain for the developers, e. g. as something
> that just doing "make htmldoc" would automatically use such extensions
> without needing to actually install the extensions.

It works for everything except the extended media book, right? Or is there
something I'm missing here? I am very much interested in having one system
for *all* our docs, but we wouldn't attempt to convert a document can't be
expressed in sphinx.

> > Even if it means still doing that ugly
> > docproc step to call kernel-doc. We can improve from there, and I
> > definitely appreciate your work on making this work with sphinx
> > extensions.
>
> I disagree: We should not cause documentation regressions by
> producing incomplete documentation and losing tables because of
> such conversion.

> The quality of the documentation after the change should be *at least*
> equal to the one we current produce, never worse.

Agreed; that's why I think the existing DocBook mechanism should stay in
place until that document will be improved by the change. But I'd rather
not hold up the entire process for one book, especially since pushing that
process forward can only help in dealing with those final problems.

> > That said, how would it work to include the kernel-doc extension in the
> > kernel source tree? Having things just work if sphinx is installed is
> > preferred over requiring installation of something extra from pypi. (I
> > know this may sound backwards for a lot of projects, but for kernel I'm
> > pretty sure this is how it should be done.)
>
> Yeah, using pypi seems an additional undesired step. Aren't there
> any way to make python to use an additional extension at the
> Kernel tree without needing to install it?

Well, sphinx has to come from somewhere too. But yes, we should be able to
carry extensions in the kernel tree. But I would still rather upstream it
(to sphinx) if possible, so we're not stuck trying to maintain it across
several sphinx releases. I don't know how volatile their interfaces are,
but it would be, in any case, the analog of an out-of-tree module...

jon

[Mauro Carvalho Chehab]

Em Wed, 4 May 2016 10:59:36 -0600
Jonathan Corbet <cor...@lwn.net> escreveu:

> On Wed, 4 May 2016 13:50:35 -0300
> Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:
>
> > Em Wed, 4 May 2016 19:13:21 +0300
> > Jani Nikula <jani....@intel.com> escreveu:
> > > I think we should go for vanilla sphinx at first, to make the setup step
> > > as easy as possible for everyone.
> >
> > Vanilla Sphinx doesn't work, as reST markup language is too limited
> > to support the docs we have. So, we should either push the needed
> > extensions to Sphinx reST or find a way to put it at Kernel tree
> > without causing too much pain for the developers, e. g. as something
> > that just doing "make htmldoc" would automatically use such extensions
> > without needing to actually install the extensions.
>
> It works for everything except the extended media book, right? Or is there
> something I'm missing here?

As far as I know, yes, but I didn't check the other documentation
to be sure.

> I am very much interested in having one system
> for *all* our docs, but we wouldn't attempt to convert a document can't be
> expressed in sphinx.

Looking from the other side, keeping only one document as DocBook
and having everything else converted to some markup solution seem
equally a bad solution, as, in the end of the day, we'll become
dependent of two different tool chains to produce document.

Also, media documentation is not just one more documentation. It is
the biggest one we have, and that has more changes than any other
documentation under Documentation/DocBook:

$ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
116
$ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
179

It also is more than twice the size of the other DocBook docs:

$ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1
82275 total
$ wc -l $(ls *.tmpl|grep -v media)|tail -1
29568 total

E. g. media corresponds to 60% of the number of patches and 73% of
the DocBook stuff.

Not converting it to the new "official" Kernel markup language
would would mean that developers that work on more than the
media subsystem would need to know two different, incompatible
dialects to produce documentation (plus kernel-doc). That sounds
messy and it is an extra penalty to media developers. Some may
even start to think that that writing bad docs would be a good thing.

> > > Even if it means still doing that ugly
> > > docproc step to call kernel-doc. We can improve from there, and I
> > > definitely appreciate your work on making this work with sphinx
> > > extensions.
> >
> > I disagree: We should not cause documentation regressions by
> > producing incomplete documentation and losing tables because of
> > such conversion.
>
> > The quality of the documentation after the change should be *at least*
> > equal to the one we current produce, never worse.
>
> Agreed; that's why I think the existing DocBook mechanism should stay in
> place until that document will be improved by the change. But I'd rather
> not hold up the entire process for one book, especially since pushing that
> process forward can only help in dealing with those final problems.

Yeah, in practice, we won't be converting everything on just one
Kernel version. Yet, we need at least an strategy and a plan that would
allow converting everything to the new markup language.

Provided that we would merge Sphinx support on 4.8, I would like to convert
the media documentation just after merging the new toolchain upstream,
in order to minimize the conversion impact and to not lose developers
and good developer's documentation in the process.

However, such work can only be started if we have a way to work with the
required extensions, as we're deciding to use a markup language that has
incomplete table support.

So, IMHO, we should have a way to run the required extensions in a
painless way before doing the merge.

> > > That said, how would it work to include the kernel-doc extension in the
> > > kernel source tree? Having things just work if sphinx is installed is
> > > preferred over requiring installation of something extra from pypi. (I
> > > know this may sound backwards for a lot of projects, but for kernel I'm
> > > pretty sure this is how it should be done.)
> >
> > Yeah, using pypi seems an additional undesired step. Aren't there
> > any way to make python to use an additional extension at the
> > Kernel tree without needing to install it?
>
> Well, sphinx has to come from somewhere too. But yes, we should be able to
> carry extensions in the kernel tree. But I would still rather upstream it
> (to sphinx) if possible, so we're not stuck trying to maintain it across
> several sphinx releases. I don't know how volatile their interfaces are,
> but it would be, in any case, the analog of an out-of-tree module...

Yeah, the best would be to put everything we need at Sphinx upstream,
but assuming that they agree today to add the extensions we need,
we would still need to carry those extensions at the Kernel tree for a
while, in order to wait for Sphinx to release a version with those
versions, and such new version is not merged back at the distributions
we use.

Thanks,
Mauro

[Jonathan Corbet]

On Wed, 4 May 2016 14:57:38 -0300
Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:

> Also, media documentation is not just one more documentation. It is
> the biggest one we have, and that has more changes than any other
> documentation under Documentation/DocBook:
>
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> 116
> $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
> 179
>
> It also is more than twice the size of the other DocBook docs:
>
> $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1
> 82275 total
> $ wc -l $(ls *.tmpl|grep -v media)|tail -1
> 29568 total
>
> E. g. media corresponds to 60% of the number of patches and 73% of
> the DocBook stuff.

These numbers are not entirely representative, I have to say. You're
ignoring the kerneldoc comments - which is what much of the "DocBook"
documents are made of, and which is the focus of much of this activity. If
you could find a way to count those, you'd get a different picture.

But I don't think that really matters; there doesn't seem to be *that* much
disagreement here.

The media book is important; we want it to be a part of the overall kernel
documentation suite and not stuck in some DocBook ghetto. I agree that we
should have an idea for a plausible path for *all* of our documentation.
But I'm also concerned about delaying this work yet again; we have
developers trying to push forward with improved documentation, and they've
had to wait a year for this stuff - so far.

Thanks,

jon

[Mauro Carvalho Chehab]

Em Thu, 5 May 2016 07:02:10 -0600
Jonathan Corbet <cor...@lwn.net> escreveu:

> On Wed, 4 May 2016 14:57:38 -0300
> Mauro Carvalho Chehab <mch...@osg.samsung.com> wrote:
>
> > Also, media documentation is not just one more documentation. It is
> > the biggest one we have, and that has more changes than any other
> > documentation under Documentation/DocBook:
> >
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l
> > 116
> > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc -l
> > 179
> >
> > It also is more than twice the size of the other DocBook docs:
> >
> > $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1
> > 82275 total
> > $ wc -l $(ls *.tmpl|grep -v media)|tail -1
> > 29568 total
> >
> > E. g. media corresponds to 60% of the number of patches and 73% of
> > the DocBook stuff.
>
> These numbers are not entirely representative, I have to say. You're
> ignoring the kerneldoc comments - which is what much of the "DocBook"
> documents are made of, and which is the focus of much of this activity. If
> you could find a way to count those, you'd get a different picture.

Yeah, if we take the big picture, I'm pretty sure that there are more
stuff written using kerneldoc. However, what is written at the
kerneldoc comments don't use DocBook markup, but its own markup
language.

Assuming that we'll keep using kerneldoc script to convert from the
Kerneldoc NANO markup, it means that any changes at the kerneldoc
backend to use DocBook/Markdown/asciidoc/reST/... won't change anything
for the developers nor will require them to actually use the new markups.
So, they can gradually start using new markups as they wish/learn, if
they want to improve the documentation. Lazy developers could just ignore
reST if they want, as everything will work as before.

However, for the files written directly in DocBook, there's no option:
at the moment the doc is converted, the developer will need to submit
patches using the new markup language. So, we need to be sure that
such transition will happen in a way that won't cause more harm than
needed. I would prefer to convert all of them at one, on a single Kernel
release (or two), in order to minimize the impact for the developers.

>
> But I don't think that really matters; there doesn't seem to be *that* much
> disagreement here.
>
> The media book is important; we want it to be a part of the overall kernel
> documentation suite and not stuck in some DocBook ghetto. I agree that we
> should have an idea for a plausible path for *all* of our documentation.
> But I'm also concerned about delaying this work yet again; we have
> developers trying to push forward with improved documentation, and they've
> had to wait a year for this stuff - so far.

Yeah, I understand.

Regards,
Mauro

[Markus Heiser]

Hi Mauro,

Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:

> Em Wed, 4 May 2016 11:34:08 +0200
> Markus Heiser <markus...@darmarit.de> escreveu:
>
>> Hi all, (hi Jonathan, please take note of my offer below)
>>
>> Am 03.05.2016 um 16:31 schrieb Daniel Vetter <daniel...@ffwll.ch>:
>>
>>> Hi all,
>>>
>>> So sounds like moving ahead with rst/sphinx is the option that should
>>> allow us to address everyone's concerns eventually? Of course the
>>> first one won't have it all (media seems really tricky), ...
>>
>> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ...

>> For this, I wrote the "flat-table" reST-directive, which adds
>> missing cells automatically:
>>
>> doc: http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table
>> source: https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py
>
> Yeah, this should address the lack of a proper way to markup cell/row
> spans, providing the additional bits for the tables we have at media.
>
> Yet, there are some issues with table conversions. See below.

> Some tables, like the one here:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html
>
> are truncated (tested with Mozilla and Chrome), and part of the information is
> lost due to that.

Not a problem of rendering. This was a bug in the magration from DocBook to reST.
You might remember that we have discussed, that some of the tables are better
marked-up as definition lists. This was (the last) one I forgot to convert to
a definition list ... I hope it was the last one, if not and you find more
or other broken parts, please inform me (on the linux-tv mailing, or direct).

--Markus--

[Markus Heiser]

Hy Jani,

Am 04.05.2016 um 18:13 schrieb Jani Nikula <jani....@intel.com>:

>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet <cor...@lwn.net>:
>>
>>> I think all of this makes sense. It would be really nice to have the
>>> directives in the native sphinx language like that. I *don't* think we
>>> need to aim for that at the outset; the docproc approach works until we can
>>> properly get rid of it. What would be *really* nice would be to get
>>> support for the kernel-doc directive into the sphinx upstream.
>>
>> No need for kernel-doc directive in sphinx upstream, later it will be
>> an extension which could be installed by a simple command like
>> "pip install kernel-doc-extensions" or similar.
>>
>> I develop these required extension (and more) within my proof of concept
>> on github ... this takes time ... if I finished all my tests and all is
>> well, I will build the *kernel-doc-extensions* package and deploy it
>> on https://pypi.python.org/pypi from where everyone could install this
>> with "pip".
>
> I think we should go for vanilla sphinx at first, to make the setup step
> as easy as possible for everyone. Even if it means still doing that ugly
> docproc step to call kernel-doc. We can improve from there, and I
> definitely appreciate your work on making this work with sphinx
> extensions.

+1

> That said, how would it work to include the kernel-doc extension in the
> kernel source tree? Having things just work if sphinx is installed is
> preferred over requiring installation of something extra from pypi. (I
> know this may sound backwards for a lot of projects, but for kernel I'm
> pretty sure this is how it should be done.)

Thats all right. Lets talk about the extension infrastructure by example:

First we have to chose a folder where we place all the *sphinx-documentation*
I recommending:

/share/linux/Documentation/sphinx

Next we have to chose a folder where reST-extensions should take place, I
would prefer ... or similar:

/share/linux/Documentation/sphinx/extensions

Lets say, you wan't to get in use of the "flat-table" extension.

Copy (only) the rstFlatTable.py file from my POC extension folder (ignore
other extensions which might be there) ...

https://github.com/return42/sphkerneldoc/tree/master/doc/extensions

Now lets say you are writing on a gpu book, it wold be placed in the folder:

/share/linux/Documentation/sphinx/gpu

In this gpu-folder you have to place the conf.py config file, needed to
setup the sphinx build environment.

/share/linux/Documentation/sphinx/gpu/conf.py

In this conf.py you have to *register* your folder with the extensions.

<SNIP conf.py> --------

import os.path, sys

EXT_PATH = "../extensions" # the path of extension folder relative to the conf.py file
sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH)))

# now import the "flat-table" extension, it will be self-registering to docutils

import rstFlatTable

<SNIP conf.py> --------

Thats all, you can run your sphinx-build command and the flat-tables in your
reST sources should be handled as common tables.

ASIDE:

You will find similar parts in your conf.py which you have created
with the sphinx-quickstart command. There, you will also find a block
looks like ...

extensions = [
'sphinx.ext.autodoc'
....
]

Don't try to add flat-table extension to this list. This list is a list
of sphinx extensions, we will use it later for other *real* sphinx
extensions.

A few words about the flat-table extension and a (future) kernel-doc one:

The flat-table is a pure docutils (the layer below sphinx) extension which
is not application specific, so I will ask for moving it to the docutils
upstream.

The kernel-doc extension on the other side is a very (very) kernel specific
application, this would never go to sphinx nor docutils upstream.

--Markus--

[Markus Heiser]

Hi Jani,

I forget to mentioning, with a local copy of my kernel-doc script:

https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc

You could do reST markup in the source code comments and extract them.

This might be a interim workaround which helps you not to edit source
code comments twice.

-- Markus--

[Markus Heiser]

Hi all, hi Jonathan,

Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:

> Em Fri, 6 May 2016 15:32:35 +0200
> Markus Heiser <markus...@darmarit.de> escreveu:
>
>> Hi Mauro,
>>
>> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:
>>
>>> Markus,
>>>
>>> Em Fri, 6 May 2016 13:23:06 +0200
>>> Markus Heiser <markus...@darmarit.de> escreveu:
>>>
>>>>

>>>> In this conf.py you have to *register* your folder with the extensions.

>>>> A few words about the flat-table extension and a (future) kernel-doc one:
>>>

>>> ...

>>>
>>>> The flat-table is a pure docutils (the layer below sphinx) extension which
>>>> is not application specific, so I will ask for moving it to the docutils
>>>> upstream.
>>>

>>> So, if I understood well, your proposal is to have a conf.py and the
>>> flat-table (plus other extensions) at the Kernel tree.
>>
>> Each book (better call it root-node) is a Sphinx-project, each
>> Sphinx-project need a conf.py file (the build configuration file)
>> in its reST-source tree.
>>
>> * http://www.sphinx-doc.org/en/stable/config.html
>>
>>> Assuming that docutils upstream receives the flat-table extension
>>> (and eventually modifies it), while the new version doesn't arrive
>>> all distros, we'll end by having some developers using the newer
>>> docutils with the extension, plus others using the in-tree one.
>>>
>>> Is there a way to specify at the conf.py what extension variant
>>> should it use, in case of both the in-tree or the docutils have
>>> the same?
>>
>> The build configuration file is a regular python file, you can
>> implement conditions whatever you want/need.
>>
>>> This could be trickier if they end by modifying the extension,
>>> but we can always backport the latest version, if they change the
>>> API.
>>
>> As far as i know, the docutils API is stable since 2002. In the
>> meantime there has been so many application build on it that
>> it is not realistic, you will see a not backward compatibly
>> change.
>>
>> The docutils project is conservative, very conservative, IMO to
>> conservative.
>>
>> Today I'am doubtful if it isn't better I would merge it sphinx
>> upstream. I have to discuss this with some maintainers, but
>> before I have to persuade myself, that all aspects are covered
>> and the implementations are robust. We are at the beginning and
>> we should not fear about every bit which could happen in the future.
>>
>> The sphinx / docutils bottom plate gives us a number of degrees
>> of freedom to find answers to question we have not yet asked. ;-)
>
> Ok. So, from what I understand, once Sphinx support is added at
> Kernel upstream, we could convert the media docbook to
> reST+flat-table extension, adding such extension either on a shared
> place or only for the media DocBook build, together with its
> conf.py.
>

Yes, in your media-conf you could decide to use a extension.

> Once it reaches upstream (either sphinx or docutils), we can
> work to make it integrate better with upstream as needed.
>
> Right?

yes, right :-)

> If so, I'm ok with merging it as soon as possible.

If we advice a merge of the flat-table directive we should
bundle this with the (to implement) "kernel-doc" directive ...

> In reST the directive might look like:
>
> <reST-SNIP> -----
> Device Instance and Driver Handling
> ===================================
>
> .. kernel-doc:: drivers/gpu/drm/drm_drv.c
> :doc: driver instance overview
> :exported:
>
> <reST-SNAP> -----

and the patches from my kernel-doc perl script to produce

reST from source code comments.

With this bundle within the kernel tree we have a good starting
point to compose reST documents from scratch and to migrate book
by book from DocBook to reST.

I insist to migrate book by book, because there are some
broken books. Broken by that, that some sources have changed
but not the corresponding documentation which use the comments
of these sources ... grap "Ooops" in the builded (xml or rst) docs.

E.g. I greped the .rst file and found the following Oops in the migrated books:

./books/mtdnand/pubfunctions-000-012.rst:13:Oops
./books/scsi/mid_layer-000-001-016-003.rst:13:Oops
./books/s390-drivers/ccw-000-004-003.rst:13:Oops
./books/device-drivers/devdrivers-000-003-048.rst:13:Oops
./books/device-drivers/devdrivers-000-003-050.rst:13:Oops
./books/device-drivers/Basics-000-001-002.rst:13:Oops
./books/device-drivers/devdrivers-000-003-031.rst:13:Oops
./books/device-drivers/Basics-000-009-032.rst:13:Oops
./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops
./books/gadget/api-000-011-005.rst:13:Oops
./books/gadget/api-000-011-009.rst:13:Oops
./books/gadget/api-000-011-007.rst:13:Oops
./books/gadget/api-000-011-003.rst:13:Oops
./books/gadget/api-000-011-011.rst:13:Oops
./books/genericirq/intfunctions-000-009.rst:13:Oops

----- Summarize ----

@Jonathan: what do you think? Should I prepare a patch
with a basic reST (sphinx) build infrastructure, including

* a folder for sphinx docs:

./Documentation/sphinx/

* flat-table & kernel-doc extension at

./Documentation/sphinx/extensions

* a patch with rst-Output for the kernel-doc perl script at

./scripts/kernel-doc

* An example document "HowTo document with reST" at

./Documentation/sphinx/kernel-doc-rst-howto

which at minimum describes the "flat-table" and "kernel-doc"
directive and the requirements for a building docs.

* a make file which fit into the kernel Makefile infrastructure (not
the one created by sphinx-quickstart).

-- Markus --

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> @Jonathan: what do you think? Should I prepare a patch
> with a basic reST (sphinx) build infrastructure, including
>
> * a folder for sphinx docs:
>
> ./Documentation/sphinx/

I'm already working on a patch series taking a different approach. I
don't think we should hide the documentation under an extra folder named
after a tool. Actually, I'm strongly opposed to that.

Instead, we should place the Sphinx stuff directly under Documentation,
and have Sphinx recursively pick up all the *.rst files. We should
promote gradually switching to lightweight markup and integration of the
documents into one system. This process should be as little disruptive
as possible.

If someone wants to convert a .txt document to .rst and get it processed
by Sphinx, it should be as simple as renaming the file, doing the
necessary edits, and adding it to a toctree. Imagine gradually
converting the files under, say, Documentation/kbuild. Why should the
.rst files be moved under another directory? They should stay alongside
the .txt files under the same directory. There's bound to be a lot of
people who'll never use Sphinx, and will expect to find the good old
plain text files (albeit with some markup) where they always were.

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 16:27:21 +0200

Maybe the best here is to add a kernel-doc-rest script, avoiding
the risk of regressions while we don't migrate all the books.

> * An example document "HowTo document with reST" at
>
> ./Documentation/sphinx/kernel-doc-rst-howto
>
> which at minimum describes the "flat-table" and "kernel-doc"
> directive and the requirements for a building docs.
>
> * a make file which fit into the kernel Makefile infrastructure (not
> the one created by sphinx-quickstart).

Works for me. Btw, if we're following the above, as the sphinx
build will be independent and won't affect the current documentation
build, then perhaps we could merge it on 4.7 merge window.

While Documentation for things like device-drivers.tmpl should of be
handled via the documentation tree, if we apply the needed
infrastructure, each subsystem maintainer would be freed to work
on the docs that are specific to their own subsystem. So, in my case,
I can handle the needed patches for the media subsystem DocBook via the
media development tree (as I currently do for
Documentation/DocBook/media).

Regards,
Mauro

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:06:49 +0300
Jani Nikula <jani....@intel.com> escreveu:

> On Fri, 06 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> > @Jonathan: what do you think? Should I prepare a patch
> > with a basic reST (sphinx) build infrastructure, including
> >
> > * a folder for sphinx docs:
> >
> > ./Documentation/sphinx/
>
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.
>
> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.

We won't avoid the need of moving things among directories, as we
have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Ok, if we put the .rst files at Documentation, we very likely reduce
the number of renames, but we'll increase the Makefile complexity,
and the risk of breakages. One alternative would be to put the sphinx
stuff on a separate Makefile, but using multiple makefiles on a single
dir is not standard at the Kernel.

> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.

Well, git will show the change as a rename, no matter if the
directory name changes or not (except if we keep the rst files
with .txt extension), but I agree with you that people will expect
to see text files at Documentation, and most will just read it without
caring to run Sphinx.

--
Thanks,
Mauro

[Markus Heiser]

Am 06.05.2016 um 17:06 schrieb Jani Nikula <jani....@intel.com>:

> On Fri, 06 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
>> @Jonathan: what do you think? Should I prepare a patch
>> with a basic reST (sphinx) build infrastructure, including
>>
>> * a folder for sphinx docs:
>>
>> ./Documentation/sphinx/
>
> I'm already working on a patch series taking a different approach. I
> don't think we should hide the documentation under an extra folder named
> after a tool. Actually, I'm strongly opposed to that.

Could you post a link to a repo? / thanks

There is no need for concurrency, let's work together on your repo.
Within my POC I realized similar building processes we will need in the
kernel sources ... where you have cascading configuration. A base
configuration which fits for all common cases and (if needed) a
*per-book* configuration.

At the end, when it comes to generate pdf books/articles, man pages
and e.g. texinfo files out of a sphinx-project you will need a build
infrastructure like this.

> Instead, we should place the Sphinx stuff directly under Documentation,
> and have Sphinx recursively pick up all the *.rst files. We should
> promote gradually switching to lightweight markup and integration of the
> documents into one system. This process should be as little disruptive
> as possible.
>
> If someone wants to convert a .txt document to .rst and get it processed
> by Sphinx, it should be as simple as renaming the file, doing the
> necessary edits, and adding it to a toctree. Imagine gradually
> converting the files under, say, Documentation/kbuild. Why should the
> .rst files be moved under another directory? They should stay alongside
> the .txt files under the same directory. There's bound to be a lot of
> people who'll never use Sphinx, and will expect to find the good old
> plain text files (albeit with some markup) where they always were.
>

Ok, I agree with you in the fact that a additional "sphinx" folder
is unrewarding.

This means (e.g.) a migrated Documentation/DocBook/gpu book should placed
in Documentation/gpu ... but don' try to merge all (Doc-)Books and .txt-files
into one sphinx project!

You will need on sphinx-project for each DocBook and one single
sphinx-project where you collect the .txt to .rst migrated files.

Am 06.05.2016 um 17:23 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:

> We won't avoid the need of moving things among directories, as we
> have lots of stuff under DocBook/ dir (btw, named after the toolchain).

Yes, it is named by the toolchain, but no one reads xml-files. Reading
text files is common.

> Ok, if we put the .rst files at Documentation, we very likely reduce
> the number of renames, but we'll increase the Makefile complexity,
> and the risk of breakages.

I don't see a great potential of breakages ... if we place every book
in a separated folder and have one project which collects the .txt files
(see above).


--Markus--

[Jani Nikula]

On Fri, 06 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
> Am 06.05.2016 um 17:06 schrieb Jani Nikula <jani....@intel.com>:
>
>> On Fri, 06 May 2016, Markus Heiser <markus...@darmarit.de> wrote:
>>> @Jonathan: what do you think? Should I prepare a patch
>>> with a basic reST (sphinx) build infrastructure, including
>>>
>>> * a folder for sphinx docs:
>>>
>>> ./Documentation/sphinx/
>>
>> I'm already working on a patch series taking a different approach. I
>> don't think we should hide the documentation under an extra folder named
>> after a tool. Actually, I'm strongly opposed to that.
>
> Could you post a link to a repo? / thanks

Very much a work-in-progress
https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx

I was hoping to polish it a bit more before showing it to the world.

> There is no need for concurrency, let's work together on your repo.
> Within my POC I realized similar building processes we will need in the
> kernel sources ... where you have cascading configuration. A base
> configuration which fits for all common cases and (if needed) a
> *per-book* configuration.
>
> At the end, when it comes to generate pdf books/articles, man pages
> and e.g. texinfo files out of a sphinx-project you will need a build
> infrastructure like this.

...

> You will need on sphinx-project for each DocBook and one single
> sphinx-project where you collect the .txt to .rst migrated files.

Surely you know more about Sphinx than I do, but I specifically would
like to include e.g. gpu documentation in the main build. I'm really
hoping we can have *additional* configuration files for special cases
(only) as needed.

[Markus Heiser]

Hi Mauro,

Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:
> Yeah, it looks better, however table truncation seem to be
> happening also on other parts, like the tables on this page:
>
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> (original table: https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> This table should contain 32 bits, but only the first 7 bits are shown
>
> and those (among others):
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
>
> Hmm... after looking more carefully, it added a horizontal scroll bar.
> That looks ugly, IMHO, and makes harder to understand its contents. The
> last one, in particular (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> is a big table on both horiz and vert dimensions. Try to read how the
> bits are packed on a random line in the middle of the table, like
> MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.

I know what you mean ;-) ... I'am also unhappy, but I will address this point
later when it goes to finish the layout.

Currently lets focus on contend and (the two)extensions.

> The table here looks weird (although it is correct):
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
> (original table: https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
>

I validated this and the other tables you mentioned above ... these are
all correct migrated ... it is 1:1 translated from DocBook ... they
might be show different to this what you know from your docbook
toolchain, because in the docbook-html you have no table grids and
wrong cellspans are not clear ... sometimes, like in the last example
you gave:

<tgroup cols="5" align="center">
<colspec align="left" colwidth="2*" />
<tbody valign="top">

a colspec might ambiguous ... so there is no clear role to migrate.

> It seems that Sphinx is assuming something like "A4 portrait" for
> the margins, while those big tables would only fit (in PDF) as
> "A4 landscape".

No, no, no ;-)

Sphinx assumes nothing about the layout, sphinx and the underlying
docutils mostly juggling with nodes and the writers in e.g. the
html-writer, outputs a clear HTML without any style but with classified
HTML tags. Styling is done in the presentation layer, in HTML, it is
done in CSS.

> I guess the better would be to not limit the right
> margin or to change it where those big tables happen, in order to
> allow PDF generation.

Generating PDF has nothing to do with generating HTML. To generate
PDF there is a other writer, the latex2e writer, which produce
LaTeX markup from which you build PDF or other printed-like medias.

>
> There are also some tables that went wrong. See the Color Sample
> Location table at:
> https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> I'm pretty sure we'll need to fix some cases like that manually.
> I didn't check, but perhaps, in this case, the DocBook were using
> empty columns just to make the table bigger. If so, this is not a
> problem with the conversion, and should be manually fixed later.

+1

Yes, lets do it manually later ... what I have in my POC is a automated
process, it is hard to consider individuals in an automatic process.
Making details *nicer* and making ambiguous markups clear is manually
done in minutes where I need hours to implement this in a automated
process.

Aside, from: http://docutils.sourceforge.net/docs/peps/pep-0258.html

Docutils Project Model -- Project components and data flow:

+---------------------------+
| Docutils: |
| docutils.core.Publisher, |
| docutils.core.publish_*() |
+---------------------------+
/ | \
/ | \
1,3,5 / 6 | \ 7
+--------+ +-------------+ +--------+
| READER | ----> | TRANSFORMER | ====> | WRITER |
+--------+ +-------------+ +--------+
/ \\ |
/ \\ |
2 / 4 \\ 8 |
+-------+ +--------+ +--------+
| INPUT | | PARSER | | OUTPUT |
+-------+ +--------+ +--------+


This is a bit simplified, because we use sphinx, which
has "builders" and sits on top of this architecture.
But it might help to see, that processes like reading,
transforming and writing are discrete.

In short: readers (the reST file reader) are creating node trees,
which are transformed by a transformer (e.g. a HTML transformer),
the writer only writes the output to a file (and copies some files
like CSS files).

If I say "HTML-writer" I address the unity off the HTML-transformer
plus the HTML-writer. In Sphinx terminus/architecture, replace the
word writer with the word "builder" ... there you have (e.g.) a
"HTML builder" and a "LaTeX builder".

--Markus--

[Mauro Carvalho Chehab]

Em Fri, 6 May 2016 18:26:10 +0200
Markus Heiser <markus...@darmarit.de> escreveu:

> Hi Mauro,
>
> Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab <mch...@osg.samsung.com>:
> > Yeah, it looks better, however table truncation seem to be
> > happening also on other parts, like the tables on this page:
> >
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html
> > (original table: https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html)
> > This table should contain 32 bits, but only the first 7 bits are shown
> >
> > and those (among others):
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html
> >
> > Hmm... after looking more carefully, it added a horizontal scroll bar.
> > That looks ugly, IMHO, and makes harder to understand its contents. The
> > last one, in particular (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html),
> > is a big table on both horiz and vert dimensions. Try to read how the
> > bits are packed on a random line in the middle of the table, like
> > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean.
>
> I know what you mean ;-) ... I'am also unhappy, but I will address this point
> later when it goes to finish the layout.
>
> Currently lets focus on contend and (the two)extensions.

OK.

> > The table here looks weird (although it is correct):
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html
> > (original table: https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html)
> >
>
> I validated this and the other tables you mentioned above ... these are
> all correct migrated ... it is 1:1 translated from DocBook ... they
> might be show different to this what you know from your docbook
> toolchain, because in the docbook-html you have no table grids and
> wrong cellspans are not clear ... sometimes, like in the last example
> you gave:
>
> <tgroup cols="5" align="center">
> <colspec align="left" colwidth="2*" />
> <tbody valign="top">
>
> a colspec might ambiguous ... so there is no clear role to migrate.

Ok, that's what I was thinking. Ok, this can be fixed later manually,
where needed. Of course one way would be to disable grids on those
tables, but I would instead fix it.

>
> > It seems that Sphinx is assuming something like "A4 portrait" for
> > the margins, while those big tables would only fit (in PDF) as
> > "A4 landscape".
>
> No, no, no ;-)
>
> Sphinx assumes nothing about the layout, sphinx and the underlying
> docutils mostly juggling with nodes and the writers in e.g. the
> html-writer, outputs a clear HTML without any style but with classified
> HTML tags. Styling is done in the presentation layer, in HTML, it is
> done in CSS.

Hmm... Then there's something deadly wrong at CSS template, as it is
shown texts only at half of my horizontal res (1920).

Probably this is the culpit:

.container { margin: 50px auto 40px auto; width: 600px; text-align: center; }

width is set to 600px, instead of using a percentage, like 100%
(or 90%).

>
> > I guess the better would be to not limit the right
> > margin or to change it where those big tables happen, in order to
> > allow PDF generation.
>
> Generating PDF has nothing to do with generating HTML. To generate
> PDF there is a other writer, the latex2e writer, which produce
> LaTeX markup from which you build PDF or other printed-like medias.

Ok.

>
> >
> > There are also some tables that went wrong. See the Color Sample
> > Location table at:
> > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html
> > I'm pretty sure we'll need to fix some cases like that manually.
> > I didn't check, but perhaps, in this case, the DocBook were using
> > empty columns just to make the table bigger. If so, this is not a
> > problem with the conversion, and should be manually fixed later.
>
> +1
>
> Yes, lets do it manually later ... what I have in my POC is a automated
> process, it is hard to consider individuals in an automatic process.
> Making details *nicer* and making ambiguous markups clear is manually
> done in minutes where I need hours to implement this in a automated
> process.

Yeah, we should not try to fix everything via auto-scripts, and
spending time right now with manual fixes will be wasted, as we need
to run it at the latest media documentation, as changes might have
happened upstream.

--
Thanks,
Mauro