Feature Request (Autodoc extension)

[Mahnoor Asghar]

Dear all,

I am currently working on auto-documenting Ironic's REST API, using the Sphinx autodoc extension. I want to be able to use docstrings to document the API, and Sphinx+autodoc fits my needs perfectly (especially considering Ironic was already using Sphinx to generate its documentation). 

There is one feature that I would like to add to the autodoc extension, though. The API method parameters are repeated across many methods, and many modules. For example, a node object is a parameter in many API calls, and its description should be the same everywhere. For this purpose, I want to put all the parameter descriptions in one file (e.g. parameters.yaml file here); and just link the key from that file in all the docstrings.

For example, documentation should be generated from the following docstring, and the parameter descriptions should be read from a file called 'parameters.yaml':

def get_all(self, node=None, resource_class=None, state=None, marker=None,
                limit=None, sort_key='id', sort_dir='asc', fields=None,
                owner=None):

"""

Lists all Allocations.

Request
-------
.. rest_parameters:: parameters.yaml
   - node: r_allocation_node
   - resource_class: r_resource_class
   - state: r_allocation_state
   - owner: owner
   - fields: fields
   - limit: limit
   - marker: marker
   - sort_dir: sort_dir
   - sort_key: sort_key

Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
   - uuid: uuid
   - candidate_nodes: candidate_nodes
   - last_error: allocation_last_error
   - name: allocation_name
   - node_uuid: allocation_node
   - resource_class: allocation_resource_class
   - state: allocation_state
   - traits: allocation_traits
   - owner: owner
   - extra: extra
   - created_at: created_at
   - updated_at: updated_at
   - links: links

"""

The format of the 'parameters' file, and the specifics of formatting are all up for discussion with the community. 

I would appreciate all of your feedback and insight, so I can develop this as effectively and as soon as possible!

Thank you and regards,

Mahnoor Asghar

[Komiya Takeshi]

Hi,

How about creating an extension like you proposed? It will work fine
if the "rest_parameters" directive generate a field list from YAML
file and given list. IMO, It's not difficult.

Thanks,
Takeshi KOMIYA

2022年1月7日(金) 17:31 'Mahnoor Asghar' via sphinx-dev
<sphin...@googlegroups.com>:

> --
> You received this message because you are subscribed to the Google Groups "sphinx-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAMJtAz_fPcb4Vns0DZdoumB17EPMFgzK%3Dkbtt6jFBD4srE9PsA%40mail.gmail.com.

[Mahnoor Asghar]

Hello,

Do you mean adding this feature to autodoc itself, or creating a new extension for this purpose?

Yes, it sounds doable!

Thank you,

Mahnoor Asghar

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAFmkQAMvBVrE7NwNaPKQs50XGc_W%3DfGxRaq512aZ3V7MB8FeYQ%40mail.gmail.com.

[Komiya Takeshi]

I think creating a new extension is better for this case.
I'm not sure the requested feature is a common feature or not.
So it would be better to implement it as a 3rd party extension at first.

Additionally, your request does not match to the concept of autodoc.
Basically, autodoc only converts the definition of python objects to
the Sphinx document. It does not modify the document itself.
It's out of scope.

Usually, it has been implemented as another extension. For example,
napoleon modifies the format of docstring (ex. numpy format to reST
format) after processing of autodoc.

Thanks,
Takeshi KOMIYA

2022年1月9日(日) 5:59 'Mahnoor Asghar' via sphinx-dev <sphin...@googlegroups.com>:

> To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAMJtAz9Z0Uc-Uf%3DuymgR6MgO0yb4g63efcnRsgXEFGe2jEgBAg%40mail.gmail.com.

[Mahnoor Asghar]

I see what you mean. 

Do other developers tend to agree with the notion that this feature does not belong with autodoc?

I think it would be a useful feature for your users, being able to place all the definitions in a file instead of repeating them, or having to set up a different extension. 

But maintaining scope is important. Perhaps I could write an extension that performs this work either before or after autodoc processes the docstrings. I would appreciate feedback to do this efficiently :)

Thank you!

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAFmkQAN9JW0tFH_b-%3D6-oD9Jf%2B3eNGkJVTJHGb7te%3DpTPMHk%3Dw%40mail.gmail.com.

[Komiya Takeshi]

Unfortunately, Sphinx team is very small and does not have enough
resources to maintain the Sphinx core.
So we need to consider carefully adding a new feature to the core. Is
it an essential feature for Python documents?
If so, it's not too late to bundle it after creating it as a 3rd party
extension.

Thanks,
Takeshi KOMIYA

2022年1月10日(月) 12:16 'Mahnoor Asghar' via sphinx-dev

> To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAMJtAz8AwO2hkZFrdpcjSvMfd4WU_PMv7Q%2BEsRa5U%3DrjF87ykQ%40mail.gmail.com.

[Mahnoor Asghar]

Since you have experience with Sphinx and its overall architecture, your opinion has got to be more well-informed than mine. To me, this is a very important feature from the point of view of Ironic. 

I am a little confused as to how I can process the output generated by autodoc in my own extension, though. 

It would be impractical for me to re-develop everything already present in autodoc, due to time constraints. Is it possible to create a fork of autodoc and add the feature?

Right now, there appear to be these options:

- Add the feature to sphinx-core

- Add the feature to autodoc

- Add the feature to an extension that processes autodoc output (before it goes to Sphinx)

- Add the feature to a new extension. However, my time constraints will be a problem if I have to re-develop everything autodoc already does.

Can we set up a meeting with the team to discuss this?

Thank you so much for your input! :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAFmkQAO813Na3OUBLjaoOqKH_iF-dAjOtfLecyup%2B8%2BZ3WkozQ%40mail.gmail.com.

[Komiya Takeshi]

>I am a little confused as to how I can process the output generated by autodoc in my own extension, though.
>It would be impractical for me to re-develop everything already present in autodoc, due to time constraints. Is it possible to create a fork of autodoc and add the feature?

You don't need to fork autodoc extension because autodoc put a
docstring of the python object as is.

For example, a function having docstring will be converted to the
following output internally:

Source code:
```
def some_func(foo, bar, baz):
"""Some wonderful feature.

Some good explanation of some_func().
The mark-ups inside the docstring will be **kept** as is.

.. image:: /path/to/image.png
"""
```

Output:
```
.. py:function:: some_func(foo, bar, baz)

Some wonderful feature.

Some good explanation of some_func().
The mark-ups inside the docstring will be **kept** as is.

.. image:: /path/to/image.png
```

After the conversion, Sphinx starts to process the generated reST
document as same as from the file.

As I commented above, autodoc does not modify docstrings at all.

To realize your idea, what you just do is to create a directive named
"rest_parameters" that generates a field list from its input.

Input:
```

.. rest_parameters:: parameters.yaml
- node: r_allocation_node
- resource_class: r_resource_class
- state: r_allocation_state
- owner: owner
- fields: fields
- limit: limit
- marker: marker
- sort_dir: sort_dir
- sort_key: sort_key

```

Output:

```
:node: explanation for r_allocation_node
:resource_class: explanation for r_resource_class
...
```

>Can we set up a meeting with the team to discuss this?

Can you speak Japanese? It's okay if you can :-)
If not, it's difficult to discuss with your team because I can't speak
English at all.
(I'm also not familiar with writing English text too...)

Thanks,
Takeshi KOMIYA

2022年1月10日(月) 18:08 'Mahnoor Asghar' via sphinx-dev

> To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAMJtAz8mSav_zNq6Ki%2Brhm2hEAbbL6Fm3SdHxzSR1DvAqDDSzQ%40mail.gmail.com.

[Mahnoor Asghar]

I see what you mean now! Thank you so much for the clear answer :)

No, unfortunately I can not speak Japanese, only English and Urdu.

Your emails are great in English, and very helpful too :)

Thank you!

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/CAFmkQANsMuxDxeaKYXVMSxyQQT1zGZ3v80MGqEwxWLo%3Ds-tmLA%40mail.gmail.com.

[Mahnoor Asghar]

Dear developers/contributors,

While developing the extension mentioned in the list, I am encountering an error that I do not understand very well.

Any clues, ideas or suggestions are welcome, to help me fix this problem. The following is the run() method that runs when my Directive is encountered.

I get an exception when adding Text nodes to a field list node, where the _html_base.py docutils writer attempts to add a class to the Text element, and ends up indexing a string.

def run(self):
        #Parse the input field list from the docstring, as a dictionary
        fieldList = {}
        fieldList = parseFieldList(self.content)
        #Read from yaml file
        paramFile = self.arguments[0]
        curPath = os.path.dirname( os.path.dirname(os.path.abspath(__file__)) )
        paramFilePath = curPath + '/' + paramFile
        yamlData = yamlParse.readFromFile(paramFilePath)
        #Nodes to return
        fieldListNode = nodes.field_list()
        fieldNode = nodes.field()
        fNamePara = nodes.Text(data = 'Parameters')
        fieldNameNode = nodes.field_name()
        fieldNameNode += fNamePara
        #Add fieldNameNode to fieldNode
        fieldNode += fieldNameNode
        #Create node to hold field body
        fieldBodyNode = nodes.field_body()
        #Substitute the descriptions
        for fieldName in fieldList:
            if fieldName in yamlData.keys():
                fieldList[fieldName] = yamlData[fieldName]["description"]
            txtNode = nodes.Text(data = fieldList[fieldName])
            fieldBodyNode += txtNode
        fieldNode += fieldBodyNode
        fieldListNode += fieldNode
        return [fieldListNode]

This is the exception output:

Exception occurred:

  File "/home/mudo/.local/lib/python3.8/site-packages/docutils/writers/_html_base.py", line 412, in set_class_on_child
    child['classes'].append(class_)
TypeError: string indices must be integers

I have tried adding different types of nodes to the field_body node, but they all recurse down onto the same error (trying to set a class on a string). Am I supposed to construct a field list differently? Is there any specific documentation that I can refer to, other than [1] and [2]?

Any pointers would greatly help! ^-^

[1]: Field Lists — Sphinx documentation (sphinx-doc.org) 

[2]: reStructuredText Markup Specification (sourceforge.io)

Thank you,

Mahnoor Asghar

[Guenter Milde]

On 2022-02-05, 'Mahnoor Asghar' via sphinx-dev wrote:

Dear Mahnoor Ashghar,

> While developing the extension mentioned in the list, I am encountering an
> error that I do not understand very well.
> Any clues, ideas or suggestions are welcome, to help me fix this problem.

> The following is the run() method that runs when my Directive is
> encountered.

> I get an exception when adding Text nodes to a field list node, where the
> _html_base.py docutils writer attempts to add a class to the Text element,
> and ends up indexing a string.

...

> I have tried adding different types of nodes to the field_body node, but
> they all recurse down onto the same error (trying to set a class on a
> string). Am I supposed to construct a field list differently? Is there any
> specific documentation that I can refer to, other than [1] and [2]?

The given references describe the reST markup language.

You may find the answer in the documentation of the Docutils document model
[3]_ [4]_. For an overview of the Docutils documentation, see [5].

Another idea is to write an rST test document that will produce the
outcome you expect from the directive (here, a field list) and inspect the
`rst2pseudoxml.py` output for the node hierarchy.

I hope this may get you started,

Günter Milde

> [1]: Field Lists — Sphinx documentation (sphinx-doc.org)

><https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>

> [2]: reStructuredText Markup Specification (sourceforge.io)

><https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#field-lists>

.. [3] The Docutils Document Tree
https://docutils.sourceforge.io/docs/ref/doctree.html#field-list

.. [4] Docutils Generic DTD
https://docutils.sourceforge.io/docs/ref/docutils.dtd

.. [5] Docutils Project Documentation Overview
https://docutils.sourceforge.io/docs/

[Mahnoor Asghar]

Thank you for the help Gunter Milde! This looks like a good direction ^-^

--
You received this message because you are subscribed to the Google Groups "sphinx-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/stlmen%24vao%241%40ciao.gmane.io.

[Mahnoor Asghar]

Dear Günter, and all,

I have looked at the links you mentioned, especially [3] and [4]. They were super helpful!

I also ran the rst2pseudoxml.py script on the input field list needed by my directive, and examined the output node structure.

The input field list looks like this:

:Field A: Field A description

:Field B: Field B description

Here is the script output:

....

<docinfo>

<field_list>

<field classes="field1">

<field_name>Field A</field_name>

<field_body>

<paragraph>Field A description</paragraph>

</field_body>

</field>

<field classes="field2">

<field_name>Field B</field_name>

<field_body>

<paragraph>Field B description</paragraph>

</field_body>

</field>

</field_list>

</docinfo>

This is exactly as I was expecting, and the problem occurs for me when I try to add a paragraph node, another text-based node, or direct text to the field_body node.

Do you have any idea what I could do differently, or is there some other approach I can try?

Any help/advice is welcome!

Thank you and regards,

Mahnoor Asghar

[Guenter Milde]

On 2022-02-07, 'Mahnoor Asghar' via sphinx-dev wrote:

> Dear Günter, and all,

This looks more like "rst2xml" output (not pseudo-XML).
Mind that the <docinfo> wrapper is an artefact from the transition of the
first field list in a document into the docinfo. You can prevent this by a
preceding paragraph or title or with the --no-doc-info command line argument
https://docutils.sourceforge.io/docs/user/config.html#docinfo-xform

> Do you have any idea what I could do differently, or is there some other
> approach I can try?

The idea is to feed rst2pseudoxml with the rST that you would use in the
input document if you want to get the same HTML as you want from
the new directive, e.g.::

My output should look

:Field A: Field A description

Second paragraph of field A.
This will be added by the new directive.


:Field B: Field B body with some `inline element` that may be added by
the directive.

Günter

[Mahnoor Asghar]

Thank you, Günter! This was very helpful. 

I had used the rst2pseudoxml.py script in the docutils/tools directory to obtain the output mentioned. 

I was able to achieve the desired result by using the make_field method of the sphinx.util.docfields.GroupedField class.

Thank you for pointing me towards the docutils tools scripts! I might be using them again to develop further functionality.

--
You received this message because you are subscribed to the Google Groups "sphinx-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/sphinx-dev/su1hlr%2417ki%241%40ciao.gmane.io.