Unwelcome surprises using Scribble on a 90,000-word novel

[Hendrik Boom]

Well, my novel draft made it through Scribble, but not intact.


(1) Some of the @ commands I use are intended to cause conditional
inclusion of their contents, dependong on a command-line arameter (which
haven't yet implemented in scribble. This is so I can keep my athor's
notes about the text in the text itself, read them while looking at
drafts, but have them removed for the final reader's copy.

I've tested this by defining one of these operations (@snip) to delete
all its contents all of the time, and that works. (Command line
dependence will come later).

But it is not possible to snip out entire sections. In particular,
@snip{
@include-section{"author-only-section.scrbl"}
}
fails, complaining that there's a nested require.

Is there some way around this?

(2) When I use include-section from the main file, the actual text in
the main file appears first, and the included files are all saved up ane
emitted after the text in the main file. I expected the sections to be
included where the include-section command was instead of being saved to
the end.

Is there some way to force immediate rather than deferred inclusion?
Text that is intended to frame a section, before and after, no
longer does. The only way around this seems to be to put the after-text
into the section itself, which is *not* the structure I want.

Of course one ot the uses I have for this is for author-only text that
talks bout problems with the section, rather than being contained in
teh section itself. But I can imagine it being useful in text intended
for the reader as well.

Wrapping text around sections makes a real difference if each section is
compiled to its own HTML page.

(3) This one isn't a real problem, but is an annoyance, since I can
instead make major changes to the source files and make them work.

When the scribble source file is already formatted in the style normally
used in English books, with a few spaces (in my case 5) starting every
paragraph and no blank line separating them, the division into
paragraphs is completely ignored and everything is run together into one
paragraph. Is this actually intended? Yes, I agree it seems to be
according to spec, because there aren't any blank lines between
paragraphs, just intentation on the first line, but is this actually
useful for any purpose?

-- hendrik

[Benjamin Lerner]

On 8/1/19 7:25 AM, Hendrik Boom wrote:

Well, my novel draft made it through Scribble, but not intact.


(1) Some of the @ commands I use are intended to cause conditional 
inclusion of their contents, dependong on a command-line arameter (which  
haven't yet implemented in scribble.  This is so I can keep my athor's 
notes about the text in the text itself, read them while looking at 
drafts, but have them removed for the final reader's copy.

I've tested this by defining one of these operations (@snip) to delete 
all its contents all of the time, and that works.  (Command line 
dependence will come later).

But it is not possible to snip out entire sections.  In particular,
   @snip{
      @include-section{"author-only-section.scrbl"}
   }
fails, complaining that there's a nested require. 

Is there some way around this?

You need to define @snip as a macro, rather than as a function, so that it can eliminate the syntax that does the require:

(define-syntax (ship stx)
  (syntax-case stx ()
    [(snip . forms)
     (if (getenv "SHOW-SNIP")
         #'(begin . forms)
         #'(begin))]))

Use this with bracket arguments, as in @snip[@include-section{blah}]

(2) When I use include-section from the main file, the actual text in 
the main file appears first, and the included files are all saved up ane 
emitted after the text in the main file.  I expected the sections to be 
included where the include-section command was instead of being saved to 
the end.

Is there some way to force immediate rather than deferred inclusion?  
Text that is intended to frame a section, before and after, no 
longer does.  The only way around this seems to be to put the after-text 
into the section itself, which is *not* the structure I want.

How are you invoking scribble? If you’re using the -html argument (rather than -htmls) or -pdf, I think it should be producing a single file in the manner you expect, unless you have sectioning commands in the included file, in which case I’m not sure what it does.

Of course one ot the uses I have for this is for author-only text that 
talks bout problems with the section, rather than being contained in 
teh section itself.  But I can imagine it being useful in text intended 
for the reader as well.

Wrapping text around sections makes a real difference if each section is 
compiled to its own HTML page.

(3) This one isn't a real problem, but is an annoyance, since I can 
instead make major changes to the source files and make them work.

When the scribble source file is already formatted in the style normally 
used in English books, with a few spaces (in my case 5) starting every 
paragraph and no blank line separating them, the division into 
paragraphs is completely ignored and everything is run together into one 
paragraph.  Is this actually intended?  Yes, I agree it seems to be 
according to spec, because there aren't any blank lines between 
paragraphs, just intentation on the first line, but is this actually 
useful for any purpose?

Since the majority of uses of Scribble seem to have been formatted-for-web technical documentation, where the formatting is non-indented paragraphs separated by blank lines, and since Racket’s syntax isn’t typically whitespace sensitive, I think this probably just made sense at the time, but you’d have to ask Matthew for more info there…

-- hendrik

​

[Hendrik Boom]

On Thu, Aug 01, 2019 at 07:41:46AM -0400, Benjamin Lerner wrote:
> On 8/1/19 7:25 AM, Hendrik Boom wrote:
>
> > Well, my novel draft made it through Scribble, but not intact.
> >
> >
> > (1) Some of the @ commands I use are intended to cause conditional
> > inclusion of their contents, dependong on a command-line arameter (which
> > haven't yet implemented in scribble. This is so I can keep my athor's
> > notes about the text in the text itself, read them while looking at
> > drafts, but have them removed for the final reader's copy.
> >
> > I've tested this by defining one of these operations (@snip) to delete
> > all its contents all of the time, and that works. (Command line
> > dependence will come later).
> >
> > But it is not possible to snip out entire sections. In particular,
> > @snip{
> > @include-section{"author-only-section.scrbl"}
> > }
> > fails, complaining that there's a nested require.
> >
> > Is there some way around this?
>
> You need to define @snip as a macro, rather than as a function, so that it
> can eliminate the syntax that does the require:
>
> |(define-syntax (ship stx) (syntax-case stx () [(snip . forms) (if (getenv
> "SHOW-SNIP") #'(begin . forms) #'(begin))])) |
>
> Use this with bracket arguments, as in |@snip[@include-section{blah}]|

That looks plausible. Except that when the snipped text is shown, it
should be in a different colour so as to identify it as snipped text.
Even if i were to modify the macro to include a colour specification
(and I don't know enough about macros or scribblt to accomplish this
myself), I'd just end up with the colour specification wrapping the
include-section command. This probably woudn't work for the same
reasons as the original snip function.

I think I'm going to have to snip section text within the section
(which will still leave a number for an empty section) or find another
mechanism for including sections.

At present, @include-section invokes require. Does it need to? Does
it actually export identifiers to the invoking scribble file? Or is
this just a convenient way of getting it to process the #lang line and
treat the included file with scribble (or, I suppose, other) syntax and
semantics?

-- hendrik

>
> > (2) When I use include-section from the main file, the actual text in
> > the main file appears first, and the included files are all saved up ane
> > emitted after the text in the main file. I expected the sections to be
> > included where the include-section command was instead of being saved to
> > the end.
> >
> > Is there some way to force immediate rather than deferred inclusion?
> > Text that is intended to frame a section, before and after, no
> > longer does. The only way around this seems to be to put the after-text
> > into the section itself, which is *not* the structure I want.
>
> How are you invoking scribble? If you’re using the |-html| argument (rather

> than |-htmls|) or |-pdf|, I think it should be producing a single file in
> the manner you expect, /unless/ you have sectioning commands in the included

> file, in which case I’m not sure what it does.

Just
scribble whole.scrbl
where whole.scrbl is the file that @include-section's all the others.

From what you say, scribble is indeed supposed to work the way I
thought it would work. So there's something peculiar about my
input.

I'll try and make a simpler case that gives the same symptoms to see
what happens.

>
> > Of course one ot the uses I have for this is for author-only text that
> > talks bout problems with the section, rather than being contained in
> > teh section itself. But I can imagine it being useful in text intended
> > for the reader as well.
> >
> > Wrapping text around sections makes a real difference if each section is
> > compiled to its own HTML page.
> >
> > (3) This one isn't a real problem, but is an annoyance, since I can
> > instead make major changes to the source files and make them work.
> >
> > When the scribble source file is already formatted in the style normally
> > used in English books, with a few spaces (in my case 5) starting every
> > paragraph and no blank line separating them, the division into
> > paragraphs is completely ignored and everything is run together into one
> > paragraph. Is this actually intended? Yes, I agree it seems to be
> > according to spec, because there aren't any blank lines between
> > paragraphs, just intentation on the first line, but is this actually
> > useful for any purpose?
>
> Since the majority of uses of Scribble seem to have been formatted-for-web
> technical documentation, where the formatting is non-indented paragraphs
> separated by blank lines, and since Racket’s syntax isn’t typically
> whitespace sensitive, I think this probably just made sense at the time, but
> you’d have to ask Matthew for more info there…

It probably *could* be done, because the indentation information does
seem to be tucked away somehere in the results from the scribble
reader, but it would require a lot of knowledge of scribble internals.

It may be easier to write a one-time indentation-to-blank line
converter and use it on the entire set of source files. I'd be sorry
to have to work from a source file format that's visaully significantly
different from the intended web page or pdf.

Similar issue with a page of bullet points written in markdown style.
Very readable, but incompatible with scribble's mechanisms. And
scribble' version is signeficantly less readable.

-- hendrik

[Robby Findler]

On Thu, Aug 1, 2019 at 12:54 PM Hendrik Boom <hen...@topoi.pooq.com> wrote:
> At present, @include-section invokes require. Does it need to? Does
> it actually export identifiers to the invoking scribble file? Or is
> this just a convenient way of getting it to process the #lang line and
> treat the included file with scribble (or, I suppose, other) syntax and
> semantics?

Each #lang scribble/base (or other scribble languages) program turns
into a module that exports `doc`. @include-section[] uses that fact to
do its work. I think changing this aspect of scribble is not likely to
work well but if you wanted some other way (that wasn't section-based)
to break up your book then you could probably build it. Alternatively,
going with the flow would entail deciding to break up files only based
on the sectioning structure (roughly). :)

Robby

[Hendrik Boom]

So I see two ways forward on the snipped include front:

(1) Write a preprocessor along the lines of the C presprocessor, but a
lot simpler, that handles the include-section[]s by actually copying
them into the preprocessed copy of everything.

or

(2) Separate the snipped sections by separating them into a separate
document. Thus the main document would no longer contain those
sections in any form whatsoever.

or

(3) Invent a completely new include mechanism. One that
explicitly calls the scribble reader on the included file and then
returns a resulting list of items as its contribution to the main text.
Some subtlety would be required here.


The first approach has the disadvantage that the line numbers in
scribble error messages would no longer be correct. Is there anything
like C's #line directive to set line numbers?

It has the advantage that the paragraph-formatting could be changed at
the same time, replacing spaces at the beginning of the line with and
extra newline.

But it really goes against the grain to apply an external preprocessor
to a language as flexible as Racket.

The second is probably the simplest workaround.

The third would likely require more knowledge of scribble internals
than I have, but it might otherwise be easy to implement.

-- hendrik

[Benjamin Yeung]

On Fri, Aug 2, 2019 at 7:25 AM Hendrik Boom <hen...@topoi.pooq.com> wrote:
>
> So I see two ways forward on the snipped include front:
>
> (1) Write a preprocessor along the lines of the C presprocessor, but a
> lot simpler, that handles the include-section[]s by actually copying
> them into the preprocessed copy of everything.
>
> or
>

> (3) Invent a completely new include mechanism. One that
> explicitly calls the scribble reader on the included file and then
> returns a resulting list of items as its contribution to the main text.
> Some subtlety would be required here.

Sorry for both being late to this discussion for possibly bringing up
something that you may already have attempted, but did you try
wrapping your "author-only" sections and defining an identifier for
it? It might be a bit of a compromise of the two items you've
referenced here, as it does need some extra code, but at least you
don't need an external tool and it's a very modest amount of
additional engineering. I had in mind something like:

On Thu, Aug 1, 2019 at 7:25 AM Hendrik Boom <hen...@topoi.pooq.com> wrote:
>
> (1) Some of the @ commands I use are intended to cause conditional
> inclusion of their contents, dependong on a command-line arameter (which
> haven't yet implemented in scribble. This is so I can keep my athor's
> notes about the text in the text itself, read them while looking at
> drafts, but have them removed for the final reader's copy.
>
> I've tested this by defining one of these operations (@snip) to delete
> all its contents all of the time, and that works. (Command line
> dependence will come later).
>
> But it is not possible to snip out entire sections. In particular,
> @snip{
> @include-section{"author-only-section.scrbl"}
> }
> fails, complaining that there's a nested require.
>
> Is there some way around this?
>

"Wrap" the "author-only-section.scrbl" material in a little bit of packaging:

#lang scribble/base
@(provide author-only-section)

@(define author-only-section
@section{ @; Or a different structure/block as appropriate
This is the content of the author-only-section.
})

----------

Then in the file where it is conditionally included:

#lang scribble/base
Pre-snip text.

@(require "author-only-section.scrbl")
@snip{@author-only-section}

Post-snip text.

----------

Of course, you can use a macro, as was alluded to above, to clean up
the `require` boilerplate. Does that do something close enough to
what you want without adding too much bulk?

Benjamin Yeung

[Hendrik Boom]

Would this mean that the author-only-section could not itself have
other @include-sections within it, being nested within a define? A
minor flaw in my case, because I don't actually do that. And I could
use the same mechanism if I needed to.

>
> ----------
>
> Then in the file where it is conditionally included:
>
> #lang scribble/base
> Pre-snip text.
>
> @(require "author-only-section.scrbl")
> @snip{@author-only-section}
>
> Post-snip text.
>
> ----------
>
> Of course, you can use a macro, as was alluded to above, to clean up
> the `require` boilerplate. Does that do something close enough to
> what you want without adding too much bulk?

It's not being done often enough to warrant a macro. It is being done
often enough to warrant a mechanism. I think this one will work.

-- hendrik

[Hendrik Boom]

On Thu, Aug 01, 2019 at 07:41:46AM -0400, Benjamin Lerner wrote:

> On 8/1/19 7:25 AM, Hendrik Boom wrote:

>
> > (2) When I use include-section from the main file, the actual text in
> > the main file appears first, and the included files are all saved up ane
> > emitted after the text in the main file. I expected the sections to be
> > included where the include-section command was instead of being saved to
> > the end.
> >
> > Is there some way to force immediate rather than deferred inclusion?
> > Text that is intended to frame a section, before and after, no
> > longer does. The only way around this seems to be to put the after-text
> > into the section itself, which is *not* the structure I want.
>
> How are you invoking scribble? If you’re using the |-html| argument (rather

> than |-htmls|) or |-pdf|, I think it should be producing a single file in
> the manner you expect, /unless/ you have sectioning commands in the included

> file, in which case I’m not sure what it does.

So I made a small case that exhibits this. Presumably I'm doing
something wrong if no one else has this problem.

Main file testmain.scrbl:

#lang scribble/base

foo
foo

@include-section["testsect.scrbl"]


bar bar


Section testsect.scrbl:

#lang scribble/base

onion
soup


The command to run scribble:

scribble testmain.scrbl


The resulting testmain.html file read in firefox and cut and pasted
into this email:

foo foo

bar bar
1

onion soup


Of course there's also a sidebar -- a table of contents with ???
instead of titles, because I didn't have any titles.

It's quite clear that the included section comes after *all* the text
in the main file, instead of being interspersed where the
@include-section occurred.

-- hendrik

[Benjamin Yeung]

Yes, that's one of the drawbacks of this way of addressing your issue.
Another is that if you change this from a section to a subsection (or
otherwise change the depth), you need to be careful about modifying
the optional file accordingly.

I hope this accomplishes what you need!

Benjamin Yeung

[Jens Axel Søgaard]

Den tor. 1. aug. 2019 kl. 13.25 skrev Hendrik Boom <hen...@topoi.pooq.com>:

(2) When I use include-section from the main file, the actual text in
the main file appears first, and the included files are all saved up ane
emitted after the text in the main file.  I expected the sections to be
included where the include-section command was instead of being saved to
the end.

Here is a typical example of how include-section is intended to be used: 

https://github.com/racket/scribble/blob/master/scribble-doc/scribblings/scribble/internals.scrbl 

The "main" file doesn't do anything, but use include other sections.
So a "solution" is to move the text you have in your main file into a separate file.

And because I was curious, I looked up the defintion of  include-section:

(define-syntax (include-section stx)
  (syntax-case stx ()
    [(_ mod)
     (with-syntax ([doc-from-mod (datum->syntax #'mod 'doc)])
       (unless (module-path? (syntax->datum #'mod))
         (raise-syntax-error #f "not a module path" stx #'mod))
       #'(begin
           (require (only-in mod [doc-from-mod doc]))
           doc))]))

So it does nothing but requiring the scribble file (which is a module exporting  doc-from-mod).
It's renamed on import (because we might need to include several documents).
Then it is simply inserted into the document at the place  include-section is used.

/Jens Axel

[Hendrik Boom]

Which makes it all the more puzzling why the included text appears to
float to the end. See the small test case I posted earlier in this
thread.

-- hendrik

>
> /Jens Axel

[Hendrik Boom]

In case it's relevant, the version of Racket and Scribble I'm using is
racket-7.4.0.1-x86_64-linux-precise.sh
downloaded about a week ago, after the large file handling resource
usage had been fixed.

And I've attached the actual html file that was generated. Hope it
survives list submission.

-- hendrik

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190802194102.hu6t3rrthydmpkb4%40topoi.pooq.com.

[Simon Schlee]

I have not completely read this thread in detail, but to me it seems like it could be useful to you, to experiment with pollen and custom tags and try to capture more semantic meaning with those tags.
I think you would have an easier time creating templates that layout the code exactly like you want, it might be some more work to transition but you would have more control over how things are rendered.
Have you already checked-out/considered using pollen?

[Hendrik Boom]

I've seen a web page about it but I haven't yet looked into it seriously.
Will look again, in more detail.

I gather that it's built upon scribble. The manual (
https://docs.racket-lang.org/pollen/ ) seems to be one meta-level above
the level of the scribble tutorials I've found. Instead of descibing
how to put together a manuscript that can produce, say, both html and
pdf (possibly via TeX) output it seems to describe how you could define
notations that could to that.

Now you need that for any kind of sophisticated use (which I gather is
what I'll be doing) but it's really nice to know what's *already*
available to produce documents. Have I somehow missed that in the
documentation bundle?

Also... pollen seems to be somewhat based on scribble.
Does is share scribble's problem of not being able to nest @include-section within another tag, like
@italic{
@include-section["other-file"]
}

-- hendrik

[bruno cuconato]

hi Hendrik,

you might want to try https://groups.google.com/forum/#!forum/pollenpub for pollen-related queries

--
bruno cuconato
(on mobile)

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190805041210.zk7aonuutslxqw7m%40topoi.pooq.com.

[Sam Tobin-Hochstadt]

What's happening here is that Scribble is designed around sections as
the primary document organization, and it treats everything in
"testmain.scrbl" as being in the initial section, and then adds an
additional section for "testsect.scrbl". If you put `@section{}` on a
line before "bar bar" the you'll see the output in the order you
expect.

Scribble documents have to be organized in terms of sections, although
that doesn't mean that you have to see the section headings in the
resulting document. Scribble provides properties for hiding these
things. Here's a pair of files that should produce the output you
want: https://gist.github.com/samth/1b0fdec97be2f8622fff54de70a1cdef

Sam

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190802194102.hu6t3rrthydmpkb4%40topoi.pooq.com.

[Hendrik Boom]

On Mon, Aug 05, 2019 at 12:05:57PM +0100, bruno cuconato wrote:
> hi Hendrik,
>
> you might want to try https://groups.google.com/forum/#!forum/pollenpub for
> pollen-related queries

And now that's presumably https://github.com/mbutterick/pollen-users/issues

-- hendrik

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/CAPWro-gSOQSFyn%3DyHcZX4Yt-VVusnF6WKXLewWYyFF9HTRCrxQ%40mail.gmail.com.