generating documentation from both reST and non-reST?

[Matthew Woehlke]

I have some documentation for a certain project. I am currently using
sphinx to generate HTML from the marked up documentation, as it seems
like a good fit; reST is a flexible markup format, sphinx lets me link
across documents, and it's very easy to add custom processing (for
example, I've tweaked it¹ into doing inter-document resolution of
`links`_), custom roles, etc.

Now I have a problem. One chunk of the documentation has been written in
reST. Another (very large) chunk cannot feasibly be written in reST, but
is written in a project-specific format. I can generate the
documentation from the reST files just fine, but I am stuck trying to
figure out how to also process the non-reST files.

It's my understanding that this should be possible with a custom class
to translate the input file into a docutils node tree. I'm not *too*
worried about that (from writing custom directives / roles, I feel like
I have a reasonable idea how to build a node tree, and I'm similarly not
too worried about the lexical processing of the file itself... at least
I'm not (currently) asking for help with any of that). What I can't
figure out however is how to tell sphinx about this class, and how to
get it to process the files.

Can anyone shed some light on this? (Bonus points for a complete example
that causes sphinx to also process all *.txt files by turning each line
of the file into a text node.)

(¹ Not just for the convenience, either; in some places, I *really* want
to use inter-documentation links in field names, which allow `links`_
but not :roles:``.)


--
Matthew

[Takayuki Shimizukawa]

Hi Matthew,

On Sat Feb 28 2015 at 4:37:57 Matthew Woehlke <mw_t...@users.sourceforge.net> wrote:

I have some documentation for a certain project. I am currently using
sphinx to generate HTML from the marked up documentation, as it seems
like a good fit; reST is a flexible markup format, sphinx lets me link
across documents, and it's very easy to add custom processing (for
example, I've tweaked it¹ into doing inter-document resolution of
`links`_), custom roles, etc.

Now I have a problem. One chunk of the documentation has been written in
reST. Another (very large) chunk cannot feasibly be written in reST, but
is written in a project-specific format. I can generate the
documentation from the reST files just fine, but I am stuck trying to
figure out how to also process the non-reST files.

It's my understanding that this should be possible with a custom class
to translate the input file into a docutils node tree. I'm not *too*
worried about that (from writing custom directives / roles, I feel like
I have a reasonable idea how to build a node tree, and I'm similarly not
too worried about the lexical processing of the file itself... at least
I'm not (currently) asking for help with any of that). What I can't
figure out however is how to tell sphinx about this class, and how to
get it to process the files.

Can anyone shed some light on this? (Bonus points for a complete example
that causes sphinx to also process all *.txt files by turning each line
of the file into a text node.)

For now, sphinx is hard-wired to reST format.

Current implementation doesn't have any reasonable API to tell sphinx another file format parser.

There is a related ticket https://github.com/sphinx-doc/sphinx/issues/825 that contains a possibility for Markdown or other format capability gist link: https://gist.github.com/tk0miya/31b553b4bd4bf987731e

 

(¹ Not just for the convenience, either; in some places, I *really* want
to use inter-documentation links in field names, which allow `links`_
but not :roles:``.)

It depends to docutils syntax.

This is not a convenient workaround:

:|foo|: bar

.. |foo| replace:: :ref:`some-label`

 

--
Matthew

 

Regards,

--

Takayuki SHIMIZUKAWA

http://about.me/shimizukawa

 

[Matthew Woehlke]

On 2015-02-27 18:14, Takayuki Shimizukawa wrote:
> On Sat Feb 28 2015 at 4:37:57 Matthew Woehlke wrote:
>> I've tweaked [sphinx]¹ into doing inter-document resolution of

>> `links`_), custom roles, etc.
>>

>> (¹ Not just for the convenience, either; in some places, I *really* want
>> to use inter-documentation links in field names, which allow `links`_
>> but not :roles:``.)
>
> It depends to docutils syntax.
> This is not a convenient workaround:
>
> :|foo|: bar
>
> .. |foo| replace:: :ref:`some-label`

Indeed. I'd found that on my own also in the process of trying to get
things to work otherwise... unfortunately I have a LOT of these (36 in
the file that's the main "offender"). This would on the one hand make
things really, really ugly, and on the other I'd have to either
duplicate the link texts or else lose having the link texts "inline"
(and they're fairly important, as the links are all three-character
identifiers). Either is fairly ugly :-).

Anyway, the link "translation" tweak works and is pretty cool :-).

--
Matthew

[Matthew Woehlke]

On 2015-02-27 18:14, Takayuki Shimizukawa wrote:

> On Sat Feb 28 2015 at 4:37:57 Matthew Woehlke wrote:
>> [long-winded story about how I want to parse both reST and another
>> format when generating documentation with sphinx]

>>
>> Can anyone shed some light on this? (Bonus points for a complete example
>> that causes sphinx to also process all *.txt files by turning each line
>> of the file into a text node.)
>
> For now, sphinx is hard-wired to reST format.
> Current implementation doesn't have any reasonable API to tell sphinx
> another file format parser.
> There is a related ticket https://github.com/sphinx-doc/sphinx/issues/825
> that contains a possibility for Markdown or other format capability gist
> link: https://gist.github.com/tk0miya/31b553b4bd4bf987731e

Ugh. Well... I'm already rolling my own sphinx-build equivalent. I'll
try to take a look at some of this stuff and figure out what parts of
sphinx I need to additionally override or whatnot to get this to work.
(I *might* even see about coming up with a proper patch for this...)

--
Matthew

[Pete]

You might consider pre-processing the *.txt files to produce .rst files.
The mapping between .txt and .rst would be your own design.
For example, Python script can parse the .txt and output .rst files into
the Sphinx document tree.  To enable this, it may be necessary to
modify the Makefile to ensure the pre-processing step is called.

This method has been successful for documentation of the NeXus scientific
data format, converting documentation embedded in XML files.

Pete

[gilberto dos santos alves]

+1 from me to prje...@gmail.com solution.

Matthew could you post some examples for all we understand if exists some kind of syntax on your txt files or what kind of txt files we are try to catch.

regards.

--
You received this message because you are subscribed to the Google Groups "sphinx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-users...@googlegroups.com.
To post to this group, send email to sphinx...@googlegroups.com.
Visit this group at http://groups.google.com/group/sphinx-users.
For more options, visit https://groups.google.com/d/optout.

--

gilberto dos santos alves
+55(11)9-8646-5049
sao paulo - sp - brasil

[Kevin Horn]

I wrote (most of) a library for generating reST files a while back, and while I wouldn't say it's "finished" by any means, you might be able to get some use out of it.

It's on bitbucket here: https://bitbucket.org/khorn/rstgen

--

Kevin Horn

[Matthew Woehlke]

On 2015-02-28 10:57, Pete wrote:
> You might consider pre-processing the *.txt files to produce .rst files.
> The mapping between .txt and .rst would be your own design.

I thought about this, but I *really* don't want to go there. For one,
I'm not sure what this wouldn't be *harder* (why try to output a
properly formatter .rst file when I can just output the docutils node
tree?). For two, I don't want to have to deal with the added build
system complexity.

I really think that just supporting additional parsers is the way to
go... and fortunately, while it does require modifying sphinx, it's
actually a fairly small and straight forward change: see
https://github.com/sphinx-doc/sphinx/pull/1747. (Mind, sphinx master is
also a lot closer to supporting this than sphinx-1.1.3-9.fc20, which
lacks support for multiple file extensions.)

The great thing about this is that it would allow using e.g. Markdown
for most of the documentation while avoiding the problem of how to
create the index... you can use reST for the index that absolutely must
have full use of reST's flexibility, and MD for things that don't.

> To enable this, it may be necessary to modify the Makefile to ensure
> the pre-processing step is called.

Ugh, who uses straight-up Make? ;-) I'm using CMake. (And Ninja. No
Makefiles for me, thank you!) Proper separation of source files and
build artifacts (which, yes, I absolutely do insist upon) is one of the
reasons preprocessing is a poor solution.

@gilberto, for various reasons I'd prefer not to post samples, but
anyway that's irrelevant. While I appreciate the offer, I don't need
help on that end. I was able to write a mostly-working parser over the
weekend. (By "mostly" I just mean that it's not done yet. It's working
for the most part but there are a few things yet that need tweaking.)

--
Matthew

[Jan Ulrich Hasecke]

Hi Matthew,

Am 27.02.2015 um 20:37 schrieb Matthew Woehlke:
> Now I have a problem. One chunk of the documentation has been written in
> reST. Another (very large) chunk cannot feasibly be written in reST, but
> is written in a project-specific format. I can generate the
> documentation from the reST files just fine, but I am stuck trying to
> figure out how to also process the non-reST files.

You can try to convert the files with Pandoc to ReST. Pandoc is a
versatile document converter, perhaps you can use it with your
"project-specific format."

juh


--
Das ZEN von Pandoc
Bücher und E-Books einfach und professionell produzieren
http://www.amazon.de/Das-ZEN-von-Pandoc-professionell/dp/1505218799/
Paperback (232 Seiten) und E-Book