evaluating Scribble for prose

[David Storrs]

tl;dr

When writing prose, what are the 'killer features' of Scribble that make people choose it over any other tool?  (Specifically in the context of prose -- Scribble is unbeatable when writing Racket documentation.)

Learning scribble seems like a big investment, and when writing prose I'm not convinced that it has more batteries-included expressive power than HTML.  For one thing, according to the docs you need to physically replace the scribble.css file to change the appearance of the HTML output and I haven't seen an option to inline it.

For point of example, I would describe the 'killer features' of HTML+CSS to be:  separation of presentation and semantics; easily perform complete transformations on appearance just by changing the CSS; store presentation commands externally or internally in a discrete segment; rich ecosystem for modification, publication, translation to other formats; and universal availability of the viewer.

Long form:

I write about 3-6,000 words per week for a shared-world story I co-author.  I'm also toying with the idea of going back and writing a sequel to one of my earlier novels.  The question then becomes:  What should I write them in?

Scribble is the obvious and unbeatable choice for technical writing about Racket, since you can evaluate code inline and the results are included for you.  For prose, I'm less clear.

I typically write raw HTML and put a stylesheet on it to get whatever effect I want.  It lets me cleanly separate content from presentation so that I can straightforwardly transform things for various media and display sizes, although when I'm publishing to a XenForo forum I need to use a simple script to translate HTML to BBCode.  Also on the down side, I have to go back and manually generate a separate version containing the commentary for my Patreon readers.

It would be nice to have a tool that:

A) Could conditionally include sections depending on environment variables or command-line switches

B) Could seamlessly generate PDF, MOBI, EPUB, and clean and valid HTML that either inlines the CSS or links to a specified CSS file as specified at publication time.  (The inline option is important for when publishing on Amazon.)

C) Handles UTF-8 correctly

I see that other people have used Scribble for non-technical purposes, so I'm curious to find out more.  Is Scribble the right tool for the job and why?

[Joel Dueck]

On Thursday, August 1, 2019 at 10:14:09 AM UTC-5, David Storrs wrote:

A) Could conditionally include sections depending on environment variables or command-line switches

B) Could seamlessly generate PDF, MOBI, EPUB, and clean and valid HTML that either inlines the CSS or links to a specified CSS file as specified at publication time.  (The inline option is important for when publishing on Amazon.)

C) Handles UTF-8 correctly

I’m not going to just come in here and scream “POLLEN!”

But, I think Pollen is ideal for this.

It does HTML very well and puts you a lot closer to the HTML/CSS "metal" than Scribble does. Your HTML/CSS output can look and be structured pretty much however you want. The “downside” is that there are no prepackaged templates, you have to be able to do HTML and CSS yourself. But that’s not an issue for you!

It doesn’t come with PDF or ebook support “built in”. But the fact that adding it is so conceptually straightforward is kind of why I ended up learning Racket. I’ve published books to both HTML and print-ready PDF from the same set of Pollen source files. MOBI and EPUB are (IIRC) mostly weird subsets of HTML/CSS so supporting them would mainly be a matter of some wrapper code in the corresponding template to package everything correctly.

And of course it’s UTF-8 friendly.

Scribble is great for Racket docs. But its document model is extremely heavy and opinionated for simple works of prose.

And talking of separating content from presentation, Pollen is kind of the ultimate framework for this. You decide what markup you want and don’t want, and you decide exactly what that markup will produce for your different output targets. This can be different for each project. Each of your Pollen projects has its own custom “markup engine” sitting right there alongside the prose sources. If you decide you want one of your tags to produce different output than before, you update your pollen.rkt file and may not even need to touch your prose sources. So your markup can be as light or as heavy as you wish, and as separate from any particular output format as you wish.

(This post seeems like it may have been specially designed to trigger me into blurting out the stock Pollen elevator pitch. If so, it worked!)

[Hendrik Boom]

On Thu, Aug 01, 2019 at 11:13:53AM -0400, David Storrs wrote:
> tl;dr
>
> When writing prose, what are the 'killer features' of Scribble that make
> people choose it over any other tool? (Specifically in the context of
> prose -- Scribble is unbeatable when writing Racket documentation.)
>
> Learning scribble seems like a big investment, and when writing prose I'm
> not convinced that it has more batteries-included expressive power than
> HTML. For one thing, according to the docs you need to physically replace
> the scribble.css file to change the appearance of the HTML output and I
> haven't seen an option to inline it.
>
> For point of example, I would describe the 'killer features' of HTML+CSS to
> be: separation of presentation and semantics; easily perform complete
> transformations on appearance just by changing the CSS; store presentation
> commands externally or internally in a discrete segment; rich ecosystem for
> modification, publication, translation to other formats; and universal
> availability of the viewer.

I too am evaluating Scribble. I realised years ago that my writing had
suffered from the use of too many word processors as technology kept
changing, and even though there were lots of conversion tools, they
didn't always work properly. I didn't want to spend time tracking down
missing italics eternally.

Also, sometimes files got edited on different systems independently.
(Blame my attantion deficit for this) I spent time doing format
conversions *and* reconciling differences by eyeball and typing.

It was plain I needed a new tool. One that could handle plain Ascii
text input (that's what my alphasmart did -- it was great for raw text
input not not so great for editing) and still handle markup for italics
and such. And one that was compatible with computer-programmer-style
revision control. (I dopted monotone as my revision control system back
then.) Word processor file formats seem to go out of their way to make
this impossible.

I ended up writing three generations of document compilers that could
accept plain Ascii text and markup, producing whatever output I wanted
at the time.

The third one, mt3, produces HTML and slightly sick TeX. And it doesn't
handle things I'm not currently using in my writing, such as tables, and
mathematical notation, and category-theoretical diagrams.

It does do markdown-style nested bullet points, which are invaluable for
planning and outlining. But that's not writing prose either.

And it's getting complicated. Each generation of my document compiler
was written as I learned what it wsa that I *reallt* needed. Each was
largely incompatible in its markup with the previous ones, but since the
bulk of what I wrote was a bunch of ordinary paragraphs containing plain
text, that wasn't much of a problem. Searchnig and change=ing by hand
in emacs was not a big deal.

But I want to do mathematics, have charts and diagrams, and so forth.

So before starting on a fourth generation, I decided to ook around.

An obvious candidate is TeX. It does not generate HTML.

Another was, of course, scribble.

You may have read my difficulties with that in a few other threads on
this mailing list in the past week or two.

First, there was excessive slowness. That was fixed a week or two ago,
making the whole project feasible. I;m now using a recent snapshot of
the development version.

Second there are problems with include-section. I can't wrap an
@include-section with anything, so it can't be conditional on a
command-line flag or anything else. This is apparently because it used
Racket's (require ...) internally, which has always to be top-level. I
gather that pollen uses the same technique, so I suspect it has the same
problem. (There do appear to be clumsy-looking workarounds involving
changes in both the included and including files)

Third, Scribble takes all my included sections and moves them to the end
of the document. I don't know why, and I've been flat-out told it
doesn't do that. I'm waiting for a fix or explanation.

Finally, there are minor notational issues. I like to start my
paragraph with indentation instead of blank lines, just so that my input
resembles the output. Easier when editing.

Of course, I could start to rewrite scribble ... But I've looked at the
codebase before (trying to figure out how to make text red), and I find
it hard to get into.

I've noticed Pollen can take markdown-stye input. (which seems
useful for my bullet-point planning and point-form sections) But
I wonder ... is that just preprocessing markdown and producing markdown?
Or is it indeed a surface syntax for pollen that gets absorbed into the
same internal notation pollen uses for non-markdown text in the dame
document.

Or is all this compatibility engineering and output generation the job
of the document author, providing ultimate flexibility with little
convenience. I'm already writing document compilers, and don't want the
equivalent of doing it yet again.

-- hendrik

> A) Could conditionally include sections depending on environment variables
> or command-line switches

Yes, this is essential so I can have author-only material interspersed
in the document.

> B) Could seamlessly generate PDF, MOBI, EPUB, and clean and valid HTML that
> either inlines the CSS or links to a specified CSS file as specified at
> publication time. (The inline option is important for when publishing on
> Amazon.)

So far I'm targetting html and pdf. Epub would be delightful.

> C) Handles UTF-8 correctly

Not essential, but easy to do sort of passably. I just recognise all
the bytes with the high but set as characters that can be in words.
Every non-Ascii Unicode character is represented as such bytes.

No, it's not perfect. It's just good enough for now.

>
> I see that other people have used Scribble for non-technical purposes, so
> I'm curious to find out more. Is Scribble the right tool for the job and
> why?

I'm not ready to use Scribble for all my writing yet. And I suspect
that Pollen has problems too, since it seems to use the same basic
architecture as Scribble. But Pollen's two-screen approach
looks quote valuable, and could mitigate difficulties with input
notation.

-- hendrik