Dynamic file order

[Lllama]

Hello all,

I'm currently building a collection of scripts for my team to use when
writing reports. The plan is that they will create a bunch of rst
files and Sphinx will turn them into a shiny PDF, or similar. Each rst
file will have a priority associated with it and I was wondering what
would be the best way to ensure the final document is ordered by this
field. I.e. High first, then medium, etc.

I've written a custom directive (not as scary as I thought) which can
be used to specify the file's priority.

So far I think the following are my best options:

1. Have a pre-build script that greps the files, extracts the
priorities and updates the index. (not ideal, as the directive won't
be used)
2. Hijack the toctree processing to do something similar to 1.
3. Create a custom writer that will process the doctrees and write
them in the correct order.

Can anyone suggest a better solution, or whether I've no hope of
achieving this?

Thanks in advance,

Felix

[Walter Wefft]

Changing the default toctree ordering is doable but may be non-trivial
depending on how you arrive at your own custom ordering. Might it be
enough to name your source files appropriately and use the :glob:
option in all toctrees? - I *think* Sphinx will eventually sort the
result of the glob, but I can't actually see in the code where this is
done atm.

Another approach might be to dynamically add the toctree directive.
Here is a snippet from some code which relates to an experimental
attempt to remove the need for authors to involve themselves with
toctrees at all, in favour of an explicit "tocindex" directive which
they are expected to add to each page. The directive (not shown)
successfully reorders the tocs but then the new ordering is
inconsistent with Sphinx's notion of next/previous. We try to fix this
by sneakily adding a hidden toctree, but as you can read in the
comment, this doesn't work because at source-read time not all the
"tocindexes" will have been gathered in general. Not saying it's a
solution for you, but maybe the idea will give you a thought.

TOCDIRECTIVE_PATTERN = re.compile('^\.\. toctree::$')
DUMMY_TOC_DIRECTIVE = """

.. toctree::
:hidden:
:glob:

*
"""

def ensure_toctree(app, docname, data):
# add a hidden toctree to any index page
if docname == app.config.master_doc or docname.endswith('/index'):
if not TOCDIRECTIVE_PATTERN.match(data[0]):
data[0] += DUMMY_TOC_DIRECTIVE
level = docname.count('/')
found_docs = app.builder.env.found_docs
alldirs = set(doc.rpartition('/')[0] for doc in
found_docs)
for dir in alldirs:
# subdirs should be added according to order in
``env.worrpt_index``
# but ``worrpt_index`` is incomplete at this point, so
we are stuck.
# this doesn't affect the TOC but it does affect how
Sphinx thinks
# things should be ordered, so next/previous will
again be borked.
# resort to updating context at page-write time?
if dir and dir.count('/') == level:
master_page = "%s/index" % dir
if master_page in found_docs:
data[0] += " %s\n" % master_page

def setup(app):
app.connect('source-read', ensure_toctree)

[Lllama]

Wow.

Okay - I think I now know a lot more about Sphinx that I wanted to but
I seem to have found a solution to my problem.

The original problem:

> > I'm currently building a collection of scripts for my team to use when
> > writing reports. The plan is that they will create a bunch of rst
> > files and Sphinx will turn them into a shiny PDF, or similar. Each rst
> > file will have a priority associated with it and I was wondering what
> > would be the best way to ensure the final document is ordered by this
> > field. I.e. High first, then medium, etc.



> > I've written a custom directive (not as scary as I thought) which can
> > be used to specify the file's priority.

Here's the directive (trimmed out some error checking for clarity):
"""
class RatingsDirective(rst.Directive):
optional_arguments = 2 ## Ideally I'd like required args...

def run(self):
env = self.state.document.settings.env
if not hasattr(env, "ratings_list"):
env.ratings_list = []

## RISK and DIFF are two dicts that contain mappings from
## "high", "med" and "low" to appropriate numerical values
## for easy sorting
risk = RISK[self.arguments[0].split(":")[1].lower()]
diff = DIFF[self.arguments[1].split(":")[1].lower()]

env.ratings_list.append(
(
risk,
diff,
env.docname,
)
)
return []

"""

> > So far I think the following are my best options:
>
> > 1. Have a pre-build script that greps the files, extracts the
> > priorities and updates the index. (not ideal, as the directive won't
> > be used)

Didn't go with this in the end.

> > 2. Hijack the toctree processing to do something similar to 1.

This is what I ended up doing. My reports will be generated as both
html and PDF (using latex). After much (much) digging I found that the
two builders use different methods for determining sort order. The
Latex
builder will pull in the pickled toctree from the index and the HTML
builder
will use cached toctree in the build environment. (I'm hoping these
are the only
two methods used).

In the end I hooked onto the "env-updated" event and rewrote the
indices there. I
used the "env-updated" event rather than "doctree-read" or "doctree-
resolved" as
this ensured all of my files had been read and that the sort order was
accurate.

Here's what I came up with:
"""

def purge_ratings(app, env, docname):
if not hasattr(env, "ratings_list"):
return
env.ratings_list = [rating for rating in env.ratings_list
if rating[-1] != docname]

def do_env_update(app, env):
if not hasattr(env, "ratings_list"):
return

## The following is required for the latex builder.
## Its sort order seems to be based on the index files rather
## than anything else

findings_doctree = env.get_doctree("index")
for toc in findings_doctree.traverse(addnodes.toctree):
toc["entries"] = [(None, unicode(entry[-1])) for entry in
sorted(env.ratings_list)]
toc["includefiles"] = [unicode(entry[-1]) for entry in
sorted(env.ratings_list)]

## Need to write the pickle back to disk, so copied this
## from the topickle method. Possible API addition...?
findings_doctree.reporter = None
findings_doctree.transformer = None
findings_doctree.settings.warning_stream = None
findings_doctree.settings.env = None
findings_doctree.settings.record_dependencies = None

for metanode in findings_doctree.traverse(MetaBody.meta):
# docutils' meta nodes aren't picklable because the class is
nested
metanode.__class__ = addnodes.meta

file_name = env.doc2path("index", env.doctreedir, ".doctree")
with open(file_name, 'wb') as f:
pickle.dump(findings_doctree, f, pickle.HIGHEST_PROTOCOL)

## Need to do this for the html versions
## All we're doing is changing the cached toc in the environment.
for toc in env.tocs["findings"].traverse(addnodes.toctree):
toc["entries"] = [(None, unicode(entry[-1])) for entry in
sorted(env.ratings_list)]
toc["includefiles"] = [unicode(entry[-1]) for entry in
sorted(env.ratings_list)]

## Register everything...
def setup(app):
app.add_directive('rating', RatingsDirective)
app.connect('env-updated', do_env_update)
app.connect('env-purge-doc', purge_ratings)
"""

So there you have it. Users can document their findings like this:

"""
My important report finding
===========================

.. rating:: rating:high diff:low

These are the details...
"""

The directive will pick up the rating, store it in the environment and
then
overwrite the tocs once all files have been read and before the writer
kicks
off.

Thought this might be useful for someone. If anyone can provide a
sanity check then
that'd be gratefully received but it seems to be working so I'm
leaving it as is until
it breaks.

Felix