How to Add Sphinx Nodes with Extension

[Konstantin Molchanov]

Hi!

I’ve been struggling with creating an extension that would parse a Swagger file and produce a well-formatted HTTPDomain-based rst document.

I’m currently using the Jinja2 workaround suggested by Eric Holscher (my extension: https://pypi.org/project/sphinxcontrib-swagger2sphinx/, Eric’s post: http://ericholscher.com/blog/2016/jul/25/integrating-jinja-rst-sphinx/).

The problem is this approach works well only with HTML output. I need to produce PDF. Although it does work with LaTeX builder, it only does so after HTML is built. Also, it has some weird side-effects like I can’t use “howto” class anymore.

So anyway, I always felt the _right_ way to do it was with the extension that operates directly on nodes. Like the one explained in the official tutorial (http://www.sphinx-doc.org/en/master/extdev/tutorial.html).

It hurts me to say, but the tutorial was barely helpful. I do have an idea on how to add docutils nodes like paragraphs, titles, admonitions, etc., but no clue on how to add Sphinx’s nodes like py:function or, more importantly in my case, http:get.

Could anyone please help me?

---

Konstantin

[Konstantin Molchanov]

When I try to add desc node[1] with desc_name and desc_content similarly to definition_list, I get the objtype missing error. Searching the web didn't help me locate and resolve this issue.

Sadly, Sphinx and docutils are horribly underdocumented. This is why any help from the Sphinx pros would be very much appreciated.

[1] http://www.sphinx-doc.org/en/master/extdev/nodes.html#sphinx.addnodes.desc

[Bram Geron]

Yeah, doing processing in the HTML writer definitely sounds like the wrong approach. 

There are (at least) three phases in Sphinx: the parsing phase, one or more processing phases, and one or more writing phases. It seems that you might be able to do everything in the parsing phase, and just immediately generate the HTTPDomain (etc.) nodes you want. That way you get all the targets already supported by those node types. Alternatively you can create your own node types, and define writer visitors for them.

I don't know specifically how to generate the appropriate nodes for HTTPDomain, but here's the Python code for an extension I made recently: https://gist.github.com/bgeron/d384a69df1834e3be3e3d65e2c50aa2c . (I still want to publish it properly sometime.) I'm not sure if you can use it as is, but perhaps it gives you more inspiration. Here's what it does.

- There is a directive class, so that Sphinx knows what code to invoke at parse time when it encounters ".. collapse::".

- When CollapseDirective.run is run, it calls the superclass, which in its turn calls recursively invokes the Sphinx parser (self.state.nested_parse) to parse the body. You won't need to do this I think, as the body of your directive will not be ReStructuredText. I also don't think you will need the node_class field, because that's used only by the BaseAdmonition superclass. You will probably parse the self.content in a different way.

- BaseAdmonition generates my node of type 'collapse', with inside it the results of the recursive parse. In lines 69-72, I generate a bunch of extra nodes inside the collapse node, just before the results of the recursive parse. You can see an example in the Gist.

- For the node types I made, I have writer visitors for all the targets I want to support (latex, html). I don't think you need a new node type, therefore you wouldn't need to make writer visitors.

Does that help a bit?

Cheers, Bram

--

You received this message because you are subscribed to the Google Groups "sphinx-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Konstantin Molchanov]

Hi, Bram!

Thanks for the reply. Although educational, your response unfortunately did not help me much. I still can't wrap my head around how Sphinx extensions work. IMHO it's way overcomplicated and at the same underexplained. Or perhaps I'm just too stupid for it :-)

Anyway, this is how I'm building my extension. I'm adding a new directive .. swagger:: which I'm replacing with a list of Docutils section nodes. I'm populating each section with more Docutils nodes. I'm not creating a new node because I can't see any reason to. So basically, I'm just appending the nodes one after another. Here's the code so far if that gives you any better understanding: http://pastebin.com/kZzvkdFS

My main issue is that for each Swagger method I'm creating a definition list item (make_definition_list_item function) whereas creating a .. http:get:: bit would be preferable. In my implementation, I have to add references to titles manually and hardcode classes to make them appear on hover, etc.

What I am looking for is a way to do something like:

method = addnodes.desc()

method += addnodes.desc_signature(text="GET /foo/bar")

method += addnodes.desc_content(method_body)

Does my thinking make any sense at all? Is there a way to do what I want? I'm feeling extremely stupid with Sphinx :-(

[Bram Geron]

Well, Sphinx is a big program and rather modular, so I think we just need more documentation. 

I don't think you can create a http:get, because I think it's not a node. If you want the behaviour of http:get, you should look at the source code for that directive, and emulate what its run method does. (That said, there might be a hacky way to generate RST source code yourself and invoke the parser on the generated RST, but that seems rather hacky and fragile.) Is it feasible to copy over some code from the source of the http:get directive?

To confirm my suspicion that http:get is not a node type, can you make some document with http:get in it and generate the pseudoxml for it? I might be wrong and http:get might be a node type after all, in which case it might be simple to generate from your directive. You can generate pseudoxml with `make pseudoxml` in the default Makefile.

Cheers, Bram

[Konstantin Molchanov]

Thanks for the make pseudoxml tip, I didn't know I could do that. Here's the pseudoxml bit for .. http:get:: /foo/bar: http://pastebin.com/73mrphfH

From XML I can see the attributes to create the desc instances with. Just tried them, and it looks like the desc is successfully created! Thanks a ton for this tip. This may be the missing clue I was looking for (given it's not anywhere in the docs, of course :-).

Also, looking at the httpdomain code was my first thought as well, but I didn't find the node definitions there [1]. I thought node creation was automatically handled by the domain API.

[1] https://bitbucket.org/birkenfeld/sphinx-contrib/src/1621e337bb236e17d9e3fe4e98976365c06cc5fb/httpdomain/sphinxcontrib/httpdomain.py?at=default&fileviewer=file-view-default

[Bram Geron]

It's indeed rather complicated, but it seems that 

- The directive class is HTTPGet

- It doesn't have a run method itself, so the run method of the superclass HTTPResource is used

- HTTPResource doesn't have a run method itself, so the run method of the superclass ObjectDescription is used

- Presumably, this is a very complicated run method that is configurable by fields, such as `doc_field_types` (from HTTPResource), and methods `needs_arglist`, `add_target_and_index`, and `get_index_text` (all from HTTPResource).

Really, ObjectDescription ought to have its own page in the Sphinx manual.

Cheers, Bram

[Konstantin Molchanov]

I was able to successfully emulate HTTPDomain nodes by analyzing the pseudoxml code. The nodes produced by my extensions are indistinguishable from the ones generated from .. http:get:: directives.

There is however one issue that hinders further progress. If I have a reference to an endpoint, e.g. :http:get:`foo/bar`, it is not resolved.

This really confuses me, because according to the logs my directive's run method is executed during source reading, and by the time references are resolved all the nodes are in place.

Another confusing thing is that if I work around the issue by connecting to "missing-reference" event and generating the proper reference manually, the warning is still in the logs. The output though is correct—I have a working reference whose pseudoxml is the same as for a regular :http:get: one.

Is there a way to resolve references in this situation? I'm sure I'm missing something here,

[Robert Lehmann]

Generating addnodes.desc gives you a definition list-like appearance but doesn't actually do anything remotely related to http:get and friends.  The better solution is probably inheriting from / creating an instance of HTTPGet, and calling its run() method.  You can modify its behaviour by setting the attributes it works on (see Directive.run), namely arguments and content.  This way you can generate a http:get directive without generating ReST source code.

To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+unsubscribe@googlegroups.com.

[Konstantin Molchanov]

Hi, Robert!

Thank you for the clarification.

Could you please show an example of using HTTPGet.run to generate the directive? I'm having trouble understanding this bit. As far as I understand it, directive is a bit of rst markup, and node is what it's translated into. So I have a single SwaggerDirective that generates a load of nodes. The concept of a directive being created within a directive messes with my mind :-(

Also, I'd rather not use HTTPGet directly because I don't know the method until it's parsed from swagger. What I need is probably create HTTPResource and set its method attribute. Is this correct?

Thanks!

[Komiya Takeshi]

Hi,

I developed sphinxcontrib-apiblueprint in this summer.

https://github.com/tk0miya/sphinxcontrib-apiblueprint

I use standard docutils nodes to represent API docs.

I only modifies domain data of httpdomains to index the APIs defined by API blueprint file.

https://github.com/tk0miya/sphinxcontrib-apiblueprint/blob/master/sphinxcontrib/apiblueprint/translator.py

I wish my source code will help you.

# I had a plan to make an extension for swagger, but I don't have enough time to do that...

Thanks,

Takeshi KOMIIYA

2016年10月5日水曜日 13時32分34秒 UTC+9 Konstantin Molchanov:

[Konstantin Molchanov]

Thanks, Takeshi! I'll certainly take a look on your code. I'd love to learn the inner workings of Sphinx to create proper extensions, but it's all a bit too complicated for me.

With my Swagger extension, I gave up and just did it via Jinja2 template https://github.com/moigagoo/sphinxcontrib-swagger2sphinx/