using scribble for everything from category theory to poetry

206 views
Skip to first unread message

Hendrik Boom

unread,
Jul 15, 2019, 5:38:22 PM7/15/19
to us...@racket-lang.org
I'm thinking of trying Scribble again now that I have a new computer.
I found it unacceptably slow years ago when I tried it on a 80,000 word novel.
I now, however, have a modern machine and it may be fast enough.

I have a few questions before I convert *everything* I'm doing to scribble.
Each of these questions are relevant to a particular planned use. The more of
them have positive answers, the more projects I'll be able to
scribblify.

(1) How does scribble handle mathematical notation? Presumably there's a hack
for when I'm generating TeX, but is there anything reasonable when generating
HTML? Mathjax is somewhat tolerable, but mathML would be nice.

(2) How can I produce category-theoretical diagrams, such as the one on top of
page 29 in section 3.7 in the pdf file
https://www.logicmatters.net/resources/pdfs/GentleIntro.pdf
Oh yes, category theorists also use different shapes of arrow
heads and tails just to challenge us computer people.

(3) Are there practical ways of including images whose source code might be
jpegs ot pngs?

(4) How can I include other scribble files into a main scribble file *without*
making it a separate section? The tutorial
https://docs.racket-lang.org/scribble/getting-started.html
mentions include-section, but I'm looking for something like just C's
#include.

(5) How can I make text conditional on the presence or absence of a
command-line parameter of the value of a global variable?

(6) How do I achieve precise, line-by-line control of indentation for poetry?

-- hendrik


Hendrik Boom

unread,
Jul 15, 2019, 8:06:51 PM7/15/19
to us...@racket-lang.org
On Mon, Jul 15, 2019 at 05:38:17PM -0400, Hendrik Boom wrote:
> I'm thinking of trying Scribble again now that I have a new computer.
> I found it unacceptably slow years ago when I tried it on a 80,000 word novel.
> I now, however, have a modern machine and it may be fast enough.

A theory why scribble is slow:

I looked at the scribble installed on my computer (which runs Devuan, a
Debian derivative) and discovered that it's all Racket source code.

So my guess is that Racket is interpreting Scribble, and the the already
interpreted Scribble is interpreting the scribble input.

Stacking interpreters is usually a recipe for sloth.

So ... is this likely the reason?

Processing an 800,000 character text file without any at signs (so that I
don't invoke any fancy scribble features) took six minutes on a modern laptop
with 16 gigabytes of RAM.

Hand-coded C processed the same file faster than I can blink my eye. Both
systems produced usable HTML. (but the hand-coded C is limited as to what it
will do. It won't handle tables of category-theoretic diagrams, for example,
and maybe Scribble can someday manage to do all this)

If this is the reason for scribble being slow, what can be done about it?
Can it, for example, be compiled to something more like machine code? (I'm
using a 64-bit intel processor.)

-- hendrik

Jack Firth

unread,
Jul 15, 2019, 8:22:35 PM7/15/19
to Racket Users
I believe the reason Scribble is slow on your file has to do with algorithms and logic in Scribble that have poor performance characteristics on large files, and nothing at all to do with the fact that Scribble is implemented in Racket. With a reproducible performance benchmark for running Scribble on large files, I'm sure a dedicated Scribble expert could do a lot to optimize the internals of Scribble and fix the performance problems you're experiencing. That said, I'm no Scribble expert.

Hendrik Boom

unread,
Jul 15, 2019, 9:12:45 PM7/15/19
to Racket Users
On Mon, Jul 15, 2019 at 05:22:35PM -0700, Jack Firth wrote:
> I believe the reason Scribble is slow on your file has to do with
> algorithms and logic in Scribble that have poor performance characteristics
> on large files, and nothing at all to do with the fact that Scribble is
> implemented in Racket. With a reproducible performance benchmark for
> running Scribble on large files, I'm sure a dedicated Scribble expert could
> do a lot to optimize the internals of Scribble and fix the performance
> problems you're experiencing. That said, I'm no Scribble expert.

Large files? Or large documents made of many small files? If the latter I can
probably manage if there's an @include mechanism.

-- hendrik
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/73523467-14eb-472c-b5c1-ea736a22e320%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Robby Findler

unread,
Jul 16, 2019, 2:47:41 AM7/16/19
to Hendrik Boom, Racket Users
I'm not sure if I'm the best one to answer these, but here are my attempts.

On Mon, Jul 15, 2019 at 4:38 PM Hendrik Boom <hen...@topoi.pooq.com> wrote:
>
> I'm thinking of trying Scribble again now that I have a new computer.
> I found it unacceptably slow years ago when I tried it on a 80,000 word novel.
> I now, however, have a modern machine and it may be fast enough.
>
> I have a few questions before I convert *everything* I'm doing to scribble.
> Each of these questions are relevant to a particular planned use. The more of
> them have positive answers, the more projects I'll be able to
> scribblify.
>
> (1) How does scribble handle mathematical notation? Presumably there's a hack
> for when I'm generating TeX, but is there anything reasonable when generating
> HTML? Mathjax is somewhat tolerable, but mathML would be nice.

That is an area that could use some help in Scribble land for now.
Something that went into mathjax and latex would be great. Maybe
others have libraries that are useful here?

> (2) How can I produce category-theoretical diagrams, such as the one on top of
> page 29 in section 3.7 in the pdf file
> https://www.logicmatters.net/resources/pdfs/GentleIntro.pdf
> Oh yes, category theorists also use different shapes of arrow
> heads and tails just to challenge us computer people.

Maybe a library on top of pict?

> (3) Are there practical ways of including images whose source code might be
> jpegs ot pngs?

Yes, sure. Any pict (or pict-convertible) can jus tbe dropped anywhere
in a scribble document that The Right Thing happens. They will be
either pngs or svg in the html output depending on how you call
scribble (e.g., via the command-line arguments).

> (4) How can I include other scribble files into a main scribble file *without*
> making it a separate section? The tutorial
> https://docs.racket-lang.org/scribble/getting-started.html
> mentions include-section, but I'm looking for something like just C's
> #include.

Maybe Racket's `include`? You can also just call functions that return
text, of course.

x.rkt:

#lang at-exp racket
(define my-favorite-paragraph @{some text here})

y.rkt:

#lang scribble/base
@(require "x.rkt")
@my-favorite-paragaph


> (5) How can I make text conditional on the presence or absence of a
> command-line parameter of the value of a global variable?

Just use regular racket for this kind of thing.

> (6) How do I achieve precise, line-by-line control of indentation for poetry?

This might require you to set up a specific thing for Scribble to
generate in each of its backends and then call into it.

https://docs.racket-lang.org/scribble/config.html

I thin you probably want to make a very small thing at that layer and
then build an abstraction on top of it in Racket/scribble to use it.
Probably others have better experience here, tho.

Robby


> -- hendrik
>
>
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190715213817.ijeovk5hs5vifgs4%40topoi.pooq.com.

tbrooke

unread,
Jul 16, 2019, 9:11:59 AM7/16/19
to Racket Users


Matthew Butterick may be able to jump in here and answer better than I can but you might want to look at Pollen https://docs.racket-lang.org/pollen/ and language that he built on top of Scribble for publishing both on the webs and in print

I am not sure about the speed since Pollen relies on Scribble but I do know that some large projects have been created with Pollen

Evelyn

unread,
Jul 16, 2019, 10:42:48 AM7/16/19
to Racket Users


On Monday, July 15, 2019 at 3:38:22 PM UTC-6, Hendrik Boom wrote:
(2) How can I produce category-theoretical diagrams, such as the one on top of
page 29 in section 3.7 in the pdf file
https://www.logicmatters.net/resources/pdfs/GentleIntro.pdf
Oh yes, category theorists also use different shapes of arrow
heads and tails just to challenge us computer people.
 
You might use https://tikzcd.yichuanshen.de/ as suggested by  
for the diagrams.

Evelyn Mitchell 

Ben Greenman

unread,
Jul 16, 2019, 11:56:07 AM7/16/19
to Hendrik Boom, us...@racket-lang.org
More answers:

On 7/15/19, Hendrik Boom <hen...@topoi.pooq.com> wrote:
> (1) How does scribble handle mathematical notation? Presumably there's a
> hack
> for when I'm generating TeX, but is there anything reasonable when
> generating
> HTML? Mathjax is somewhat tolerable, but mathML would be nice.

For TeX, I tell Scribble exactly what characters to put in the document.
The same idea might work for mathML.

https://docs.racket-lang.org/scribble-abbrevs/index.html#(def._((lib._scribble-abbrevs%2Flatex..rkt)._~24))

The scribble-math package might be a better choice:
https://docs.racket-lang.org/scribble-math/index.html

> (2) How can I produce category-theoretical diagrams, such as the one on top
> of
> page 29 in section 3.7 in the pdf file
> https://www.logicmatters.net/resources/pdfs/GentleIntro.pdf
> Oh yes, category theorists also use different shapes of arrow
> heads and tails just to challenge us computer people.

You probably don't want to use Racket/Scribble for this, but here's a
pict / ppict
version of the diagram. (I don't know how to change the arrowheads.)

https://gist.github.com/bennn/1523efe1e6759c67c6f16118aa6543d5

> (3) Are there practical ways of including images whose source code might be
> jpegs ot pngs?

Yes, @image{file.png}

> (4) How can I include other scribble files into a main scribble file
> *without*
> making it a separate section? The tutorial
> https://docs.racket-lang.org/scribble/getting-started.html
> mentions include-section, but I'm looking for something like just C's
> #include.

https://groups.google.com/d/msg/racket-users/1cMufGgwtf0/-zTwzAKZAgAJ

> (5) How can I make text conditional on the presence or absence of a
> command-line parameter of the value of a global variable?

Text can depend on an if-statement:

@(if (f x) @elem{this} @elem{that})


> (6) How do I achieve precise, line-by-line control of indentation for
> poetry?

One way is to use @verbatim{....} or @codeblock{....}

But you can also play with linebreaks and hspace:
https://gist.github.com/bennn/b936b6caf8534b1a11b1bfa7de4e245c

Hendrik Boom

unread,
Jul 18, 2019, 12:29:00 PM7/18/19
to Racket Users
On Mon, Jul 15, 2019 at 09:12:42PM -0400, Hendrik Boom wrote:
> On Mon, Jul 15, 2019 at 05:22:35PM -0700, Jack Firth wrote:
> > I believe the reason Scribble is slow on your file has to do with
> > algorithms and logic in Scribble that have poor performance characteristics
> > on large files, and nothing at all to do with the fact that Scribble is
> > implemented in Racket. With a reproducible performance benchmark for
> > running Scribble on large files, I'm sure a dedicated Scribble expert could
> > do a lot to optimize the internals of Scribble and fix the performance
> > problems you're experiencing. That said, I'm no Scribble expert.
>
> Large files? Or large documents made of many small files? If the latter I can
> probably manage if there's an @include mechanism.

Did some measurements.

800K of text, represented as 80 scribble files each of 10K bytes,
each included using @include-section from one file:
7 seconds.

800K of text, represented as one scribble file (the concatenation of
the 80 files used above):
4 minutes and 29 seconds.

Not immediately prohibitive, but it's painful to repeatedly make a
correction in scribble source, recompile, and view the result of the
change in an interactive session.

-- hendrik
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190716011242.amzacpoljvkaq4ue%40topoi.pooq.com.

Matthew Flatt

unread,
Jul 18, 2019, 12:52:32 PM7/18/19
to Hendrik Boom, Racket Users
At Thu, 18 Jul 2019 12:28:53 -0400, Hendrik Boom wrote:
> Did some measurements.
>
> 800K of text, represented as 80 scribble files each of 10K bytes,
> each included using @include-section from one file:
> 7 seconds.
>
> 800K of text, represented as one scribble file (the concatenation of
> the 80 files used above):
> 4 minutes and 29 seconds.
>
> Not immediately prohibitive, but it's painful to repeatedly make a
> correction in scribble source, recompile, and view the result of the
> change in an interactive session.

Thanks! This appears to be an algorithmic problem in the macro
expander, and Racket v6.12 (with the old expander) doesn't seem to have
this problem. I'll continue to investigate.


Matthew

Matthew Flatt

unread,
Jul 18, 2019, 2:25:26 PM7/18/19
to Hendrik Boom, Racket Users
> At Thu, 18 Jul 2019 12:28:53 -0400, Hendrik Boom wrote:
> > Did some measurements.
> >
> > 800K of text, represented as 80 scribble files each of 10K bytes,
> > each included using @include-section from one file:
> > 7 seconds.
> >
> > 800K of text, represented as one scribble file (the concatenation of
> > the 80 files used above):
> > 4 minutes and 29 seconds.
> >
> > Not immediately prohibitive, but it's painful to repeatedly make a
> > correction in scribble source, recompile, and view the result of the
> > change in an interactive session.

I've pushed a repair that corrects the kind of behavior illustrated
above, at least for my own example. I think there's a good chance that
it will correct performance for your specific example. (The repair is
pushed to the development repo, so it should be available in the next
round of snapshots in a little under 24 hours from now).

Hendrik Boom

unread,
Jul 22, 2019, 10:42:32 AM7/22/19
to Racket Users
Thank you.

Now I have to figure how to install this on my system. The Racket and
Scribble I currently have are installed from the Devuan (in this case
equivalend to Debian) package. How to install scribble from I presume
develoment source so as to relace the Debian Scribble. Persumably purge
the package and install Racket and Scribble completely from source?
Presumably there are instructions online somewhere? Or is there an
easier route?

-- hendrik

Ben Greenman

unread,
Jul 22, 2019, 9:25:39 PM7/22/19
to Racket Users
You should be able to keep the current Racket,
download a snapshot build,
and run the `..../bin/drracket` inside the snapshot

https://www.cs.utah.edu/plt/snapshots/

Hendrik Boom

unread,
Jul 23, 2019, 12:05:16 PM7/23/19
to Racket Users
So I would install one of the snapshots and use it instead of my
installed Devuan package. And that presumably means I won't need the
Devuan package and can urge it.

Assuming of course that the Ubuntu build is comatible with my system.

Which I'll find out by trying it.

-- hendrik

>
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CAFUu9R7mnyoeUwP%2B%3DmAFDJv%2BgCZ_OQ7UHDDA80zErrSo_mzQGw%40mail.gmail.com.

Hendrik Boom

unread,
Jul 23, 2019, 3:01:58 PM7/23/19
to Racket Users
On Thu, Jul 18, 2019 at 12:25:22PM -0600, Matthew Flatt wrote:
New timing, using the snapshot I downloaded and installed today,
racket-7.4.0.1-x86_64-linux-precise.sh

80 scribble files of 10K each: one or two seconds
One 800K file: four or five seconds

Much improved, even for the many smaller files case.

Thank you.

-- hendrik

Hendrik Boom

unread,
Jul 27, 2019, 11:05:52 PM7/27/19
to Jens Axel Søgaard, Racket Users
On Tue, Jul 16, 2019 at 03:30:37PM +0200, Jens Axel Søgaard wrote:
> Den man. 15. jul. 2019 kl. 23.38 skrev Hendrik Boom <hen...@topoi.pooq.com
> >:
>
> > (1) How does scribble handle mathematical notation? Presumably there's a
> > hack
> > for when I'm generating TeX, but is there anything reasonable when
> > generating
> > HTML? Mathjax is somewhat tolerable, but mathML would be nice.
> >
>
> The package scribble-math provides easy access to Mathjax and KaTex.
>
> https://docs.racket-lang.org/scribble-math/index.html
>
> There is also a way to write figures with Asymptote (which only compiles
> the Asymptote
> snippets, when changes has been made).
>
>
> > (2) How can I produce category-theoretical diagrams, such as the one on
> > top of
> > page 29 in section 3.7 in the pdf file
> > https://www.logicmatters.net/resources/pdfs/GentleIntro.pdf
> > Oh yes, category theorists also use different shapes of arrow
> > heads and tails just to challenge us computer people.
> >
>
> It's hard to compete with TikZ, but I am currently working on extending
> MetaPict with
> nodes and edges ala TikZ. The diagram in the example could be drawn
> with something like:
>
> (def A (text-node "A"))
> (def B (text-node "B" #:right-of A))
> (def C (text-node "C" #:below B))
> (def D (text-node "D" #:right-of C))
> (draw A B C D
> (edge A B #:label "f")
> (edge A C #:label "g ∘ f")
> (edge B C #:label "g")
> (edge B D #:label "h ∘ g")
> (edge C D #:label "h"))

This looks potantially very useful.

>
> I'll send you an email, when the new code is in a reasonable state.

I'd be happy to look at the code even *before* it's ready for use.
(I could probably learn something fom it.)
I hope it can be used to produce output for both HTML and TeX.
Will TeX expessions wourk in the text-nodes and labels?

-- hendrik

>
> /Jens Axel
Reply all
Reply to author
Forward
0 new messages