sphinx-click: Cross-referencing Click Groups and Commands

[Tomer]

Hello,

I'm trying to use Sphinx to automatically generate documentation for a command line tool written in Python using Click.

I'm using the sphinx-click plugin for this, but I'm having trouble figuring out how to cross-reference specific types of functions - namely commands and command groups.

I can reference options using :option:`group_name-command_name -option_name`, and I can reference environment variables using :ref:`DisplayText <group-command-option-envvar>` (doesn't seem to work without supplying a name, as written here), but despite doing much trial and error I can't find how to reference command or command groups.

The only solution I've found is HTML-specific, by using a direct link like `DisplayText <group-page.html#group-command>`_ - or `DisplayText <#group-command>`_ if it's on the same page - but that's obviously not a very good solution.

Seeing as the table of contents entry does reference these things, I figure there's got to be a way to get these references, so I'm hoping someone here might have an answer for me.

Being able to read the .doctree files to see what all the label names are might be enough, but I don't know how to do that either.

Any help would be appreciated.

Thanks in advance!

[Stephen Finucane]

On Sun, 2022-04-03 at 03:59 -0700, Tomer wrote:
> Hello,
>
> I'm trying to use Sphinx to automatically generate documentation for a command
> line tool written in Python using Click.
> I'm using the sphinx-click plugin for this, but I'm having trouble figuring
> out how to cross-reference specific types of functions - namely commands and
> command groups.

Hey,

Just FYI, sphinx-click is an extension, which means the sphinx-users group
generally isn't the best place to ask questions like this. You're better off
using the issue tracker for the project. With that said, I'm the maintainer of
sphinx-click and subscribe to this list so perhaps you've got lucky :)

> I can reference options using :option:`group_name-command_name -option_name`,
> and I can reference environment variables using :ref:`DisplayText <group-
> command-option-envvar>` (doesn't seem to work without supplying a name, as
> written here), but despite doing much trial and error I can't find how to
> reference command or command groups.
> The only solution I've found is HTML-specific, by using a direct link like
> `DisplayText <group-page.html#group-command>`_ - or `DisplayText <#group-
> command>`_ if it's on the same page - but that's obviously not a very good
> solution.

This is actually called out in the documentation [1]:


Programs

Sphinx currently does not allow you to cross-reference programs. See Sphinx
issue #880 for more information.

So this is an issue with Sphinx itself that we can't fix. What we _could_ do is
add a custom anchor like we now do for environment variables which you could
then reference. It would be quite trivial to do and I'd happily welcome a PR. It
just hasn't been done yet.

Cheers,
Stephen

[1] https://sphinx-click.readthedocs.io/en/latest/usage/#cross-referencing

PS: A more involved PR would involve adding custom roles for all of these so
that we could avoid using Sphinx's (somewhat broken) standard domain
roles/directives entirely. That's a lot more work though and I suspect this
would require some good working Sphinx knowledge.

>
> Seeing as the table of contents entry does reference these things, I figure
> there's got to be a way to get these references, so I'm hoping someone here
> might have an answer for me.
> Being able to read the .doctree files to see what all the label names are
> might be enough, but I don't know how to do that either.
>
> Any help would be appreciated.
> Thanks in advance!
>
>

> Disclaimer
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may be
> unlawful.
> --
> You received this message because you are subscribed to the Google Groups
> "sphinx-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sphinx-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sphinx-users/0e383cdc-06ec-4327-8ec4-6b8213daee4bn%40googlegroups.com
> .

[Tomer]

Hi Stephen,

Thank you for taking the time to reply.

I was aware that this was a Sphinx group, but I didn't find anywhere better to ask, and I thought that if someone at least knew how I could read the .doctree files that would also help, and that does sound relevant to this group.

I also thought I might luck out and find someone who knows about sphinx-click specifically, so I'm glad that worked out. =)

Since I had a question rather than an actual issue I didn't think the issue tracker was the right place for it. Is that normally an acceptable place for general questions?

Regarding your answer:

I did see that in the documentation, but I didn't know that the commands and groups were considered "programs."

I misunderstood that to only refer to the top level of the Click application and thought that the documentation merely didn't specify anything about groups and commands in general.

I'm very new to Sphinx so unfortunately not yet well versed in its terminology.

Having those custom anchors would be very nice for sure.

I don't know if I'll have the time to get into this to actually implement it myself but I'll tell you if I do (or open an issue and say it there).

Thank you for maintaining the project - it's most definitely very useful!

Since I managed to reach you here, I hope you don't mind if I ask a couple of more questions. =P

I'd like to generate documentation for a CLI tool with multiple command groups, which would show quite a bit of information.

To this end I'd like to split it into separate pages - similar to how the Click --help display works, where it shows you short help text for the groups if you run it without any other arguments, and then shows things specific to a group if you run it with  "group --help".

I haven't found a built-in way to do this, so my current solution is to generate an .rst page for each group and put a ".. click::" directive there.

Not ideal, but should work. If you have a better suggestion I'd love to know.

However, I don't know what to do in order to display a short summary of the commands for each group.

Ideally I'd like to put three of these directives one after the other, with their :nested: options set to none, short and full respectively. Doing this actually works somewhat decently, but some of the information repeats itself (namely the contents of the none form shows up in all of them).

Is there a way to make the directive not list the program name and description?

Another thing I'd like to do is be able to have the titles of the commands in the short form reference their anchors. Even if it's done manually in the form of entering a link to the HTML page and section, that would still be nice to have.

Is there a way to modify the output of that directive?

Thanks again,

Tomer.

[Stephen Finucane]

On Mon, 2022-04-04 at 06:38 -0700, Tomer wrote:
> Hi Stephen,
> Thank you for taking the time to reply.
>
> I was aware that this was a Sphinx group, but I didn't find anywhere better to
> ask, and I thought that if someone at least knew how I could read the .doctree
> files that would also help, and that does sound relevant to this group.
> I also thought I might luck out and find someone who knows about sphinx-click
> specifically, so I'm glad that worked out. =)
> Since I had a question rather than an actual issue I didn't think the issue
> tracker was the right place for it. Is that normally an acceptable place for
> general questions?

It depends on the project but I'm happy for you to do that, personally. I
suspect you'll be told if something is not considered appropriate.

> Regarding your answer:
> I did see that in the documentation, but I didn't know that the commands and
> groups were considered "programs."

Yeah, this is Sphinx terminology. We're just borrowing directives and roles from
the standard domain [1]. Each subcommand is considered a separate program in
Sphinx lingo.

> I misunderstood that to only refer to the top level of the Click application
> and thought that the documentation merely didn't specify anything about groups
> and commands in general.
> I'm very new to Sphinx so unfortunately not yet well versed in its
> terminology.

If you think that the documentation could be clarified then by all means, I'd
happily accept PRs for this too (even WIP PRs - I can finish them up myself).

> Having those custom anchors would be very nice for sure.
> I don't know if I'll have the time to get into this to actually implement it
> myself but I'll tell you if I do (or open an issue and say it there).

No worries. I'm unlikely to have time myself to draft this entirely but I'd be
happy to drag something over the line if you or someone else proposed a solid
WIP PR at some point.

> Thank you for maintaining the project - it's most definitely very useful!
> Since I managed to reach you here, I hope you don't mind if I ask a couple of
> more questions. =P
>
> I'd like to generate documentation for a CLI tool with multiple command
> groups, which would show quite a bit of information.
> To this end I'd like to split it into separate pages - similar to how the
> Click --help display works, where it shows you short help text for the groups
> if you run it without any other arguments, and then shows things specific to a
> group if you run it with  "group --help".
> I haven't found a built-in way to do this, so my current solution is to
> generate an .rst page for each group and put a ".. click::" directive there.
> Not ideal, but should work. If you have a better suggestion I'd love to know.

This is how I'd do this also. Again, it's possible that sphinx-click could
autogenerate these pages for you in a similar vein to how Sphinx's sphinx-apidoc
tool will generate .rst pages containing 'autodoc' directives [2], but it
doesn't currently do this. Another option is to enable nested documentation
generation (using the 'nested' argument), but this would put everything on a
single page. You seem to have already explored this.

> However, I don't know what to do in order to display a short summary of the
> commands for each group.
> Ideally I'd like to put three of these directives one after the other, with
> their :nested: options set to none, short and full respectively. Doing this
> actually works somewhat decently, but some of the information repeats itself
> (namely the contents of the none form shows up in all of them).
> Is there a way to make the directive not list the program name and
> description?

Not at this time, no. It's very much an all or nothing thing. I hadn't really
envisioned further customization than what the existing arguments ('prog',
'nested', 'commands') allows.

> Another thing I'd like to do is be able to have the titles of the commands in
> the short form reference their anchors. Even if it's done manually in the form
> of entering a link to the HTML page and section, that would still be nice to
> have.
> Is there a way to modify the output of that directive?

Again, modification of the output isn't currently possible beyond what those
three options allow for. It would probably be useful to open an issue with
screenshots of what you're trying to achieve (and why) so we can see whether
this is worth doing.

>
> Thanks again,
> Tomer.

Hope this helps,
Stephen

[1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#the-standard-domain
[2] https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html

[Tomer]

Hi Stephen,

Pardon the late reply; I wanted to make some decisions and try things out before getting back to you.

I've forked the sphinx-click repository and made a few modifications locally.

Namely what I've added is:

• Anchors for groups and commands in the same dash-separated form as the rest (group, group-command).
• Extra anchors for options and arguments to conform to the same standard (`group-command-option`, `group-command-arg`)
  • Namely if you have an option with multiple names, e.g. -v, --verbose for command `log` in the group `run`, you'll get both `run-log-v` and `run-log-version`.
• A new :hide-header: flag which hides the title, description and usage. If used with the ":nested: none" setting, this would produce an empty output, so in that specific case I made it only remove the title.
• A new nesting value which I very poorly named ":nested: complete" for lack of a better idea (suggestions are welcome). It's basically like the ":nested: full" setting but it also includes the short descriptions from ":nested: short" before it shows the full details. See attached screenshot below for an example.

I only have it working with my little experimentation project though. Haven't done any proper testing yet.

Aside from that, I've written a small module to facilitate generating references using the added anchors to the documentation from within the docstrings of the CLI commands themselves.

I don't know if that's of any interest to you - if so you're welcome to read on.

I'll first explain why.

I saw that if I put something like the following in a docstring for an example "starter order-soup" command:

    Soup goes great with :ref:`meat <main-order-meat>`, you know (just be sure to specify a good :option:`dish <main order-meat -dish>`).

It works and generates the references.

However, as you would expect, if you then ran the CLI tool and typed "cli starter order-soup --help", you would get exactly the above text, which is naturally not something you want to see in a CLI help message.

So I wrote something that wraps Click's command decorators, parses the docstrings of each command and replaces the references with suitable text.

I made it so that you can set some environment variable to say that you're currently building the documentation - if so then it replaces the text with references, otherwise it removes everything but the name.

(By the way, do you know if there's a way to detect whether we're running Sphinx or the actual CLI without using an environment variable like that? That'd be nicer.)

Since I transform the text in both instances I can be more flexible in format, so I made it so that you can type any of the following:

    :type_of_reference:`Display Name Here <reference_path>`

    :type_of_reference:`reference_path`

    `Display Name Here <reference_path>`

    `reference_path`

and you'll get a reference in Sphinx-generated documents and only the display names in the CLI itself.

(The "type of reference" bit is to let you choose to use :option: instead of :ref: since that only works for options but also does some nice formatting.)

If any of this is of interest to you I'd love to share it, so please tell me if so.

Aside from that, regarding what you said, I do think the documentation can be clearer about that specific point, and also I believe there's a copy/paste error there in the Cross-referencing section where it says:

    Specifically, it uses the program, program, and program directives.

Thanks again for your help,

Tomer.

[moses Ageyo]

KcushTrends .com

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "sphinx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sphinx-users/f1df6215-9eb3-4b01-aa57-d31339995c9dn%40googlegroups.com.

[Stephen Finucane]

On Wed, 2022-04-06 at 07:14 -0700, Tomer wrote:

Hi Stephen,

Pardon the late reply; I wanted to make some decisions and try things out before getting back to you.

I've forked the sphinx-click repository and made a few modifications locally.

Namely what I've added is:

• Anchors for groups and commands in the same dash-separated form as the rest (group, group-command).
• Extra anchors for options and arguments to conform to the same standard (`group-command-option`, `group-command-arg`)
  • Namely if you have an option with multiple names, e.g. -v, --verbose for command `log` in the group `run`, you'll get both `run-log-v` and `run-log-version`.
• A new :hide-header: flag which hides the title, description and usage. If used with the ":nested: none" setting, this would produce an empty output, so in that specific case I made it only remove the title.

These all sound reasonable. I'd be happy to review a PR with these. It doesn't need to be initially complete, assuming you'd like feedback before investing time into writing tests.

• A new nesting value which I very poorly named ":nested: complete" for lack of a better idea (suggestions are welcome). It's basically like the ":nested: full" setting but it also includes the short descriptions from ":nested: short" before it shows the full details. See attached screenshot below for an example.

:nested: fuller ?

I only have it working with my little experimentation project though. Haven't done any proper testing yet.

Aside from that, I've written a small module to facilitate generating references using the added anchors to the documentation from within the docstrings of the CLI commands themselves.

I don't know if that's of any interest to you - if so you're welcome to read on.

I'll first explain why.

I saw that if I put something like the following in a docstring for an example "starter order-soup" command:

    Soup goes great with :ref:`meat <main-order-meat>`, you know (just be sure to specify a good :option:`dish <main order-meat -dish>`).

It works and generates the references.

However, as you would expect, if you then ran the CLI tool and typed "cli starter order-soup --help", you would get exactly the above text, which is naturally not something you want to see in a CLI help message.

So I wrote something that wraps Click's command decorators, parses the docstrings of each command and replaces the references with suitable text.

I made it so that you can set some environment variable to say that you're currently building the documentation - if so then it replaces the text with references, otherwise it removes everything but the name.

(By the way, do you know if there's a way to detect whether we're running Sphinx or the actual CLI without using an environment variable like that? That'd be nicer.)

Since I transform the text in both instances I can be more flexible in format, so I made it so that you can type any of the following:

    :type_of_reference:`Display Name Here <reference_path>`

    :type_of_reference:`reference_path`

    `Display Name Here <reference_path>`

    `reference_path` 

and you'll get a reference in Sphinx-generated documents and only the display names in the CLI itself.

(The "type of reference" bit is to let you choose to use :option: instead of :ref: since that only works for options but also does some nice formatting.)

If any of this is of interest to you I'd love to share it, so please tell me if so.

This sounds helpful, though it's tough to say how useful it would be without seeing the code. Feel free to submit a WIP pull request for review. Try to keep it separate from the other items above so we can review it in isolation.

As an aside, the typical way people seem to want to do this is using the form feed (\f) character to hide Sphinx-specific stuff from click output. You need a recent version of click to support this but it otherwise seems like a good call. More information at https://github.com/click-contrib/sphinx-click/issues/56

Another option would be to write  a plugin for click that would transform help texts and remove the Sphinx-specific stuff using something like https://pypi.org/project/rst2txt/. This would definitely be more difficult but it's not impossible.

Aside from that, regarding what you said, I do think the documentation can be clearer about that specific point, and also I believe there's a copy/paste error there in the Cross-referencing section where it says:

    Specifically, it uses the program, program, and program directives.

Good spot. Fixed.

Hope this helps,

Stephen

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "sphinx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sphinx-users/f1df6215-9eb3-4b01-aa57-d31339995c9dn%40googlegroups.com.

[Tomer]

Hey Stephen,

Thanks for your reply,

I'll draft what I have into a PR hopefully tomorrow or the next day.

I would love some feedback, yes. Considering how little I still know about all this, I'm sure there are better ways of doing what I did.

:nested: fuller, while clear, sounds a bit silly to me. XD

Maybe "verbose"?

Well, I'll include it in the PR and you're welcome to decide for yourself if you want to add it and what to call it if so.

I didn't know about the form feed character, that's a useful tip.

I don't think it would solve the issue, however, as that seems to be used for truncating everything that follows, so I wouldn't be able to use that to change inline references.

What I made seems to be working decently enough for now, at least.

Considering it modifies Click objects, it might be a good idea to make it into a plugin like you said. I actually didn't know Click supported plugins. I'll look into that.

One more thing I've added to my forked repo is a ":post-process: path.to.module:function_name" option that lets you inject code to post-process generated nodes for each command.

In terms of amount of change to sphinx-click, there isn't much there - just loading the module and then calling the specified function on the list of nodes at the end of the _generate_nodes() function.

But adding it allowed me to add per-command customization. I don't know how much something like that will be used by other people, but personally I've found it to be very useful.

I also have a working version of the documentation generation tool now.

Also without proper testing so far, unfortunately... this entire thing I'm doing is one big POC so I need to get it into presentable form and see that everything I have in mind is possible before I can take the time to do the rest of it.

Thanks,

Tomer.

[Stephen Finucane]

On Tue, 2022-04-12 at 08:33 -0700, Tomer wrote:

Hey Stephen,

Thanks for your reply,

I'll draft what I have into a PR hopefully tomorrow or the next day.

Looking forward to it 🥳

I would love some feedback, yes. Considering how little I still know about all this, I'm sure there are better ways of doing what I did.

:nested: fuller, while clear, sounds a bit silly to me. XD

Maybe "verbose"?

I was trying to find a superlative of full. It's fuller or fullest and I suspect we should reserve the latter for future 😅️ verbose also works though if you really hate fuller

Well, I'll include it in the PR and you're welcome to decide for yourself if you want to add it and what to call it if so.

I didn't know about the form feed character, that's a useful tip.

I don't think it would solve the issue, however, as that seems to be used for truncating everything that follows, so I wouldn't be able to use that to change inline references.

Yeah, it's certainly different to what you're proposing here.

What I made seems to be working decently enough for now, at least.

Considering it modifies Click objects, it might be a good idea to make it into a plugin like you said. I actually didn't know Click supported plugins. I'll look into that.

One more thing I've added to my forked repo is a ":post-process: path.to.module:function_name" option that lets you inject code to post-process generated nodes for each command.

In terms of amount of change to sphinx-click, there isn't much there - just loading the module and then calling the specified function on the list of nodes at the end of the _generate_nodes() function.

But adding it allowed me to add per-command customization. I don't know how much something like that will be used by other people, but personally I've found it to be very useful.

I've no issues with hooks. It just needs to be (very) well documented and somewhat maintainable (i.e. no spaghetti code). Unit tests are probably a necessity too, though that might be difficult since our only tests right now are for the formatting code, not the loading code or anything else...

I also have a working version of the documentation generation tool now.

Also without proper testing so far, unfortunately... this entire thing I'm doing is one big POC so I need to get it into presentable form and see that everything I have in mind is possible before I can take the time to do the rest of it.

All sounds good to me.

Cheers,

Stephen

[Tomer]

Alright, I've created a PR for the features I've added. =)

I don't hate "fuller;" it just seems strange to me because "full" sounds like a binary thing to me, but since "fuller" and "fullest" are actual words, I guess it doesn't have to be binary. 🤔

I've left it with the original name; feel free to call it whatever you think fits best.

The added hook is, as I've said, a very small change to the code, so not much spaghettibility there.

I reused the module loading code that was already present and invoked the loaded function at the end of _generate_nodes().

But yes, documentation is a must.

I can provide a usage example if that'll help.

Let me know what you think.

- Tomer.