Custom scribble renderers and xref information

[Alexis King]

I’ve been playing on and off with writing a custom scribble renderer for a blog, and my experiments have mostly been fine, but I am very confused about xref information. I want to use a URL fragment format for my blog that is different from the one used by the Racket documentation, but I would also like to be able to generate external links to the Racket documentation. My first idea was to use `resolve-get/ext?` to see if a reference was external and handle it differently, but I soon realized I have no idea what “external” means there.

Initially, I assumed that if I called `render` from `scribble/render` with multiple parts in the `docs` argument, references between the parts would be considered external. However, this does not appear to be the case. On the other hand, if I reference something in The Racket Reference and I pass `#:xrefs (list (load-collections-xref))`, those references are considered external. Strictly speaking, that is what I want, so I am not unhappy with the result, but that discovery led me to other questions:

1. I probably want each of my blog posts to be its own “document,” since they should have their own namespace of tags, but passing multiple values to `render` seems to use one namespace. I interpret that to mean that I should be doing something else, but in that case, I don’t think I know how to properly set up a context where two posts can recursively reference one another. Do I need to invoke `traverse`/`collect`/`resolve`/`render` manually?
   
   
2. To generate links to the Racket documentation with the right fragment format, should I use `xref-tag->path+anchor`, using the default value for `#:render` to use the built-in HTML renderer? And does it make sense to override `collect-part-tags`/`collect-target-element` to use my own collect info format so I can distinguish references to my blog posts from references to the Racket docs?
   
   
3. I would like to avoid re-rendering every single one of my blog posts when just one of them changes, which seems possible, since `raco setup` appears to do dependency tracking. However, I wouldn’t know where to begin if I wanted to do that dependency tracking myself.

I took a look at the code for the part of `raco setup` that renders scribble documentation, but it seems to do a lot of work, and I don’t really understand it. That said, I’m guessing I don’t actually need something nearly that complicated. I’m mostly just interested in understanding how the pieces fit together. Is the idea that I could use `#:info-out-file` to serialize information about provided tags to disk, then just use GNU Make or something similar to track a dependency on those serialized xref files and pass them to `#:info-in-file`? And if I wanted to have mutually recursive documents, would I need to run them until reaching a fixpoint?

Thanks,

Alexis

[Matthew Flatt]

Yes, I think you're arriving at the right conclusions here.

If I remember correctly, "external" means "from a different run of
`render`", which normally means different documents --- but not if you
give multiple documents to `render` at once. Since you want to be able
to render blog posts separately, different blog posts should be
`render`ed separately, and they'll count as external to each other.

Making mutual references work is the job of info-in and info-out files.
Document A's info-out file is the info-in file for any document that
needs to reference document A. Yes, you have to run to a fixed point
where none of the info-out/info-in files change. That requires a fancy
`make` rule, indeed. :)

Running individual passes (instead of using the `render` function) is
probably *not* the way to go.

To distinguish references to your own document and render them
differently, I think you're right about overriding `collect-...`
methods to generate your own format for entries. That's on the frontier
of what we've tried to make configurable, though, so it may or may not
work out as intended.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/E77B36F9-65E5-4922-AB34-3FE0ECE3
> 689C%40gmail.com.

[Alexis King]

Thanks for the prompt reply!

> On Nov 17, 2019, at 11:15, Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> Making mutual references work is the job of info-in and info-out files.
> Document A's info-out file is the info-in file for any document that
> needs to reference document A. Yes, you have to run to a fixed point
> where none of the info-out/info-in files change. That requires a fancy
> `make` rule, indeed. :)

Yes, perhaps I’ll try to avoid the need for recursive references for now. :) I do have two related followup questions, though. First, how does `raco setup` know what the dependencies are in the first place, so it knows what it needs to rerender? I’ve noticed it seems to do things in at least two passes—“running” the documentation before “rendering” it—is that somehow involved?

Second, I’d like to better understand how “tag prefixes” work, since they seem to be the mechanism by which tags from different documents are distinguished. What I don’t really understand is what the resolution logic actually is or where it happens. The documentation on tags says this:

> The prefix is used for reference outside the part, including the use of tags in the part’s tags field. Typically, a document’s main part has a tag prefix that applies to the whole document; references to sections and defined terms within the document from other documents must include the prefix, while references within the same document omit the prefix.


What part of the pipeline implements this prefix-sensitive lookup behavior? Is it the `resolve-get` family of functions, so I can just put things in the `tag-prefixes` field of a `part` during the collect phase and everything else will happen automatically? Or do I need to do something more myself? Also, I don’t completely understand the way the resolution works… is it possible for there to be some kind of “ambiguous match,” where two different tag definitions are equally-specific matches for a given reference? If so, what happens in that scenario? (And if not, why not?)

[Hendrik Boom]

On Sun, Nov 17, 2019 at 06:15:15PM +0100, Matthew Flatt wrote:
> Yes, I think you're arriving at the right conclusions here.
>
> If I remember correctly, "external" means "from a different run of
> `render`", which normally means different documents --- but not if you
> give multiple documents to `render` at once. Since you want to be able
> to render blog posts separately, different blog posts should be
> `render`ed separately, and they'll count as external to each other.
>
> Making mutual references work is the job of info-in and info-out files.
> Document A's info-out file is the info-in file for any document that
> needs to reference document A. Yes, you have to run to a fixed point
> where none of the info-out/info-in files change. That requires a fancy
> `make` rule, indeed. :)

One thing I've used in the past is a conditional copy command.

condcopy from to

first compares the from and to files to see if they are the same.
If so, it does nothing. In particular, it does not chamge the timestamp
on 'to' that make uses.
If they are different, it copies 'from' to 'to'.

So if changes were made to the input of a ommand that venerates 'from'
that don't make a difference (except for changing from's modified timestamp,
no copy will take place and the timestamp on 'to' will not change.

I used this to avoid unnecessary recompilation of generated code.
But I imagine it would be useful in approximating a cirular makefile.

-- hendrik

[Matthew Flatt]

At Sun, 17 Nov 2019 12:12:15 -0600, Alexis King wrote:
> I do have two related followup questions, though. First, how does
> `raco setup` know what the dependencies are in the first place, so it
> knows what it needs to rerender? I’ve noticed it seems to do things
> in at least two passes—“running” the documentation before “rendering”
> it—is that somehow involved?

Yes, more or less. Unless I've forgotten something, it would be ok to
start with "rendering" and then use as many "re-rendering" passes as
needed. But when starting from scratch, it's typically going to take a
couple of passes to get cross-references resolved, in which case
actually rendering the documentation the first time around would take
even longer. There's probably room for improvement through heuristics
that choose whether to use a "running" pass or not.

Dependencies are determined by keeping track of what searches the
cross-reference database are made and keeping track of which documents
end up supplying results for those searches. For example, if a document
links to `cons` from `racket/base`, then that's supplied by the
Reference, so that means a dependency for the document on the
Reference. Those dependencies are recorded by `raco setup` in
"in.sxref" files.

> Second, I’d like to better understand how “tag prefixes” work, [...]

> What part of the pipeline implements this prefix-sensitive lookup behavior?
> Is it the `resolve-get` family of functions, so I can just put things
> in the `tag-prefixes` field of a `part` during the collect phase and
> everything else will happen automatically?

Yes, that's right.

Some searches can involve multiples tries until there's a hit. That
happens with the search implemented in `scribble/racket` for linking to
bindings, where searching tries a module, then the module that the
first re-exports from (if any), and so on. In that case, there's a way
of recording all the keys that were tried, so that dependencies can be
updated when something starts supplying one of the intermediate keys.

> Also, I don’t completely understand the way the resolution works… is
> it possible for there to be some kind of “ambiguous match,” where two
> different tag definitions are equally-specific matches for a given
> reference? If so, what happens in that scenario? (And if not, why
> not?)

A warning is reported by `raco setup` when the same key is defined by
multiple documents; it inspects cross-reference information
specifically to look for those duplicates. The Scribble layer by itself
only warns about multiple definitions of a key within a document.

(Like many warnings, it would have been better to make that one an
error at the `raco setup` level, but it seemed too stringent
originally.)