Convert Makefile to command line input

49 views
Skip to first unread message

Joshua Sullivan (Verb Ext.)

unread,
May 14, 2019, 11:37:19 AM5/14/19
to sphinx-users
Hello all, I am currently working on a project that that specifically is asking me to not use a Makefile to build all of the dependencies because there entire repo uses bazel to build files instead. For this reason I figured the easiest way to do this to start off would be to convert Sphinx's current MakeFile to command line inputs and just write this into a short python script. I have zero knowledge when it comes to MakeFiles to begin with, although the Sphinx MakeFile doesn't seem overly complicated just by looking at it. I was wondering if someone would be kind enough to help me out with converting this file to a few short commands I can run through the terminal.

# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS    =
SPHINXBUILD   = sphinx-build
SOURCEDIR     = .
BUILDDIR      = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)


Thank you very much in advance. Your solution will help me a ton!

Gert van Dijk

unread,
May 14, 2019, 12:22:42 PM5/14/19
to sphinx...@googlegroups.com
On Tue, May 14, 2019 at 5:37 PM Joshua Sullivan (Verb Ext.) <joshuas...@verbsurgical.com> wrote:
Hello all, I am currently working on a project that that specifically is asking me to not use a Makefile to build all of the dependencies because there entire repo uses bazel to build files instead. For this reason I figured the easiest way to do this to start off would be to convert Sphinx's current MakeFile to command line inputs and just write this into a short python script.

That's quite a coincidence - soon, we (the company I work for) will release a project to the public that uses Sphinx with Bazel, using Bazel rules (Apache 2 licensed). This means that we currently run 'bazel build docs' and it will correctly build Sphinx documentation without the need for any dependency to install for users (except Python on the host for Bazel Python roles I believe, the rest will be installed by Bazel like it does for other toolchains too).

To get back to your question - I think your approach won't help you with Bazel for most use cases of Sphinx. You can invoke a Python script or any other executable instead of a Makefile, but that does not make it Bazel compatible. Basically, Bazel requires everything to be deterministic in and out with specified input/output labels without touching source files. To be able to also feed generated-other-Bazel-rules files to Sphinx's single-input-directory and change options in configuration-by-conf.py was the major pain to overcome.
I like Bazel a lot, and I like Sphinx a lot, but it was a bit hard hard to unify their designs.

Roughly speaking we have 1) a custom sphinx_main.py that invokes sphinx.cmd.build.main() injecting/snooping some arguments, 2) some glue in conf.py that consumes parameters via Bazel status files and environment variables that we need to pass, 3) output from other applications generating RST input files, 4) rules_sphinx/def.bzl that orchestrates everything.

Our solution is currently far from perfect:
* Python 2 only, because of Python-pip limitations with Bazel I could not tackle yet.
* No incremental builds - if Bazel detects a change on any input, a full Sphinx-rebuild of the docs will be triggered.
* No proper autodoc support (yet), because in a Bazel scenario you want Bazel to select the toolchain etc., not Sphinx. We currently build RST files from the sources with other Bazel rules and feed them to Sphinx separately in a temporary input directory.
* It's quite monolithic tightly integrated in our project, not something you can import separately yet.
* HTML target only for now.

I'll post a link in this thread once it's released to the public and I hope it will be helpful to get it kickstarted for you as well. I can't promise anything on the timeline, but I think it will be a matter of days rather than weeks. :-)

If there's interest for it, I hope that in the future it could evolve to a separate project in the long list of Bazel rules projects [1], e.g. 'rules_sphinx'.

HTH

Joshua Sullivan (Verb Ext.)

unread,
May 14, 2019, 12:56:48 PM5/14/19
to sphinx-users
This sounds exactly like what we are looking for for the most part! I do have a few clarification questions to ask if you don't mind answering them just to make sure I'm understanding the capabilities and limitations of your companies solution in regards to the soon to be released version correctly.

  1. Your implementation will only support Python 2?
    1. We just migrated completely over to Python 3.5 last week...
  2. Your implementation will not support the autodoc functionality?
  3. Will be able to generate documention with a simple bazel build <filename/directoryname>?
  4. Will have to modify sphinx's current conf.py file in order for it to work with your package?
  5. Will this eliminate the need for __init__.py files in every sub-directory?
  6. Lastly, can you explain a little more about what you mean when you said, "It's quite monolithic tightly integrated in our project, not something you can import separately yet." and what you were referring to here.
I appreciate you quick response to my original question! I am very excited to hear that there will soon be a solution for this, as I have not seen anything out there quite like what you have described to me and it is exactly the solution we are looking for by the sounds of it! Please let me know when this solution is available and I will be very happy to test this out with our project!!

Thank you again!!

Gert van Dijk

unread,
May 14, 2019, 2:19:30 PM5/14/19
to sphinx...@googlegroups.com
On Tue, May 14, 2019 at 6:56 PM Joshua Sullivan (Verb Ext.) <joshuas...@verbsurgical.com> wrote:
This sounds exactly like what we are looking for for the most part! I do have a few clarification questions to ask if you don't mind answering them just to make sure I'm understanding the capabilities and limitations of your companies solution in regards to the soon to be released version correctly.

  1. Your implementation will only support Python 2?
    1. We just migrated completely over to Python 3.5 last week...
As far as understand the only limitation for us are the Python-pip rules which are unable to invoke Pip with Python 3 correctly in our situation. My last attempt was showing Python 3 in Python runtime (e.g. print sys.version), but the pip ran by Bazel to fetch the dependencies could only find Sphinx up to 1.8.x and none of the other Python-3-only packages. Of course we would have strongly preferred to use Python 3, but for now we proceeded with the native Bazel way and hoping to get it sorted upstream soon (it may have been already, haven't checked in a month or so). The GitHub repository lists numerous issues with pip [1], unfortunately.
Are you using Bazel with Python 3.5 in that project? I feel like Python is not a first class citizen in Bazel, to be honest.
  1. Your implementation will not support the autodoc functionality?
Well, actually I'm not sure, but presumably not without extending more of the Bazel rules as we currently have. IIUC, autodoc will try to do Python-level imports to obtain the docstrings, but Bazel runs everything in a strict sandbox. This means that you will have to find the magic to find the correct path to append to sys.path or actually Python-depend on the sources you want to run autodoc on (and they will be 'installed' as other Python dependencies). However, if your current project is already fully 'Bazelified', those deps/usages should be fairly trivial.
  1. Will be able to generate documention with a simple bazel build <filename/directoryname>?
Ultimately, yes, that's how we do it! (Nit: Bazel works with labels/targets, not like plain files like Make.)

$ bazel build docs
INFO: Invocation ID: f366be17-a6aa-41b8-b8a4-017fe020147a
INFO: Analyzed target //docs:docs (64 packages loaded, 3671 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /home/user/.cache/bazel/_bazel_user/9a5697af4b4898056e1adadcd9c31af8/sandbox
Target //docs:docs up-to-date:
  bazel-bin/docs/html
INFO: Elapsed time: 8.841s, Critical Path: 4.94s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 3 total actions

  1. Will have to modify sphinx's current conf.py file in order for it to work with your package?
Depends on how much you are currently accessing from conf.py and custom extensions. If you're running commands or reading environment variables, you're going to need magic glue - Bazel does not pass the anything down, as everything is deterministic and heavily sandboxed. For us it was even challenging getting the git commit hash in the documentation.
  1. Will this eliminate the need for __init__.py files in every sub-directory?
Not sure what you're referring to? That's just Python 3.3+ behaviour as far as I understand [2]?
  1. Lastly, can you explain a little more about what you mean when you said, "It's quite monolithic tightly integrated in our project, not something you can import separately yet." and what you were referring to here.
It's not so portable. We made quite some assumptions to get started quickly as for what we need. In the current shape it's more like you could use it as inspiration on how to do it for your project.
Think of limitations like hardcoded paths, opinionated options passed to sphinx-build, not relying on doctree states, etc. Despite all that I believe it's a nice starting point.
 
I appreciate you quick response to my original question! I am very excited to hear that there will soon be a solution for this, as I have not seen anything out there quite like what you have described to me and it is exactly the solution we are looking for by the sounds of it! Please let me know when this solution is available and I will be very happy to test this out with our project!!
 
I'm glad to hear you're interested in it. Are you familiar with Bazel? If not, are you able to pair with a developer who is in the context of your project? I think it will be either very useful to the extent of required, depending on how complicated your project is regarding the documentation.

By the way, I totally forgot to link to some resources where we have started to look at in the beginning. Neither of the two projects I found had a well-shaped solution for us and we had to cherry-pick the ideas from several places.
* Drake (no clue what it is, just got some inspiration from there): [3]
* Envoy, very similar use case as ours, fully Bazel-powered project, docs together with API docs (Protobuf), but they're using plain shell scripts :-( [4]
* However, Envoy has a sort-of nice way of producing autodoc-like content from Protobuf definitions [5] which we used and integrated with the rest to combine it *all* in a single 'bazel build docs'.

Hope to update soon with the actual sources so you could have a try. :-)

HTH

Joshua Sullivan (Verb Ext.)

unread,
May 14, 2019, 3:04:46 PM5/14/19
to sphinx-users
These answers were very help! Thank you again, I will continue to check back on this string for any updates that you post. I am pretty new to Bazel but I do have several people on my team that I am sure would be will to assist if I run into any issue or have any questions once your package has been released. Thank you for taking the time today to respond to my inquiry! It has been very useful and I am sure the team will be pleased to hear the news. 

Gert van Dijk

unread,
May 16, 2019, 12:35:40 PM5/16/19
to sphinx...@googlegroups.com
We've published the project today! https://github.com/TulipSolutions/tecl

The following steps "should work" and demonstrates a fully integrated and reproducible build of Sphinx docs with Bazel (along with some custom Protobuf-to-RST documentation logic which may not be so relevant for those not using Protobuf).
  1. Install Bazel (tested with current latest stable release 0.25.1)
  2. $ git clone https://github.com/TulipSolutions/tecl && cd tecl
    (HEAD is @ 21db316 at the time of writing.)
  3. $ bazel build docs
    (output in bazel-bin/docs/html)
Relevant files for the Sphinx-Bazel integration I believe are:
  • bazel/rules_sphinx/*
  • docs/README.md
  • docs/BUILD.bazel
As mentioned earlier in this thread, this is in an alpha state, it's working for us, but it's got several rough edges and assumptions that should be improved for broader public use.

We'd be very happy to hear if there's more interest in developing this further - and of course we'd also be happy to receive contributions of course.

Enjoy! :-)
Reply all
Reply to author
Forward
0 new messages