imp: a modern typesetting system (prototype and brainstorming)

[Christoph Hack]

Hi Gophers,

Ever so often I use pdflatex to write something and get very annoyed while doing so. Don't get me wrong, LaTeX is a great and very powerful tool and I have used it excessively. For example, I have once written a service that generates a paper on a weekly basis (see [3] for an example), but doing the paper layout within TeX was very difficult (I ended up writing some Go code that runs pdflatex multiple times and some TeX hacks that measure the output of nearly everything before tweaking and composing the final layout in a final run). The missing Unicode support in LaTeX still causes occasional troubles, I wasn't able to fulfill all wishes from the designer (e.g. custom fonts) and it was quite time consuming to define the layout within TeX.

Therefore I sat down yesterday and started writing my own typesetting system. An early prototype can be found here [1]. The current implementation supports TrueType and OpenType (including Type1) fonts, encodes all glyphs directly (to avoid encoding problems and to further utilize all features offered by those fonts) and is currently able to produce this output [2]. Please note, that you have to provide your own fonts, since I haven't found any usable print fonts that are freely available yet.

Most of the stuff is still hard-coded and I haven't settled on a plan to continue. One obvious approach might be to use HTML and CSS as input. Writing a CSS layout engine is probably very time consuming, but even a partial implementation might be useful - the main goal isn't rendering complex websites after all. One advantage of a CSS (or CSS like) engine is that you have a full in-memory representation of the content including hundreds of CSS properties. The engine can try different layouts (within the given constraints) and might even use techniques like linear programming to find an optimal layout for the given content.

CSS isn't optimal for paged media, and I don't really like the CSS3 content generation module though. Generated content is a huge part in such a typesetting system - just think about headers, footers, chapter-title pages, table of content, etc. The macro expansions approach used by TeX on the other hand is terrific. It's easier to write than HTML, you can easily add your own commands and you can hook into every aspect of the layout generation by overwriting internal commands. The disadvantage here however is the missing in-memory representation of the content. TeX offers some hacks around the problem, but advanced automated layouts (e.g. multi-column layouts with floating images that spawn multiple columns for example) aren't possible and even simple layouts don't look optimal.

One idea might be to try to combine both approaches, i.e. using macros and macro expansion to add CSS like properties / constraints to an in-memory representation of the content and try to find an optimal layout in a later step. I haven't really thought about it, but the idea itself sounds promising. What do you think? Do you happen to know any good resources that might be helpful (e.g.  layout engines in general, page layout based on linear programming techniques, etc.)? What features would you like to see in an modern TeX alternative? Do you think that the CSS 3 flexbox model might be a good start for an layout engine? (having boxes with content that can flow in any direction as fundamental primitive might be very useful for other languages)

Best Regards,

Christoph

[1]: https://github.com/tux21b/imp

[2]: http://tux21b.org/public/imp-example1.pdf

[3]: http://tux21b.org/public/leupold-2012-33.pdf

[opennota]

Are you aware of lout?

[Christoph Hack]

Thanks for the tip opennota. I have just read "An Expert’s Guide to the Lout Document Formatting System" and I am very fascinated by the galley mechanism and how it can be used to define layouts spawning multiple pages with content areas, columns, footnotes, etc. elegantly.

On Sunday, April 20, 2014 6:02:23 AM UTC+2, opennota wrote:

Are you aware of lout?

[Michael Jones]

By the way, Mark Summerfield's Programming in Go is set using lout.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Hotei]

I'm wondering what version of Lout Summerfield used.  I didn't see go in Lout 3.40's prg2lout.c file (a programming language specific code formatter).

Lout's source code is interesting.  Looks like a good candidate for conversion to go :-)

[Dobrosław Żybort]

For me the most important would be good support for source code highlighting (with unicode symbols in it - never worked for me in latex... - and proper source code pagination)

Best regards,
Dobek

[gta]

On Sunday, 20 April 2014 18:19:11 UTC+2, Hotei wrote:

I'm wondering what version of Lout Summerfield used.  I didn't see go in Lout 3.40's prg2lout.c file (a programming language specific code formatter).

 

Lout's source code is interesting.  Looks like a good candidate for conversion to go :-)

Indeed, cannot wait until the c transpiler is ready :)

[Bakul Shah]

On Sat, 19 Apr 2014 13:41:50 PDT Christoph Hack <tux...@gmail.com> wrote:
> Hi Gophers,
>
> Ever so often I use pdflatex to write something and get very annoyed while
> doing so. Don't get me wrong, LaTeX is a great and very powerful tool and I
> have used it excessively. For example, I have once written a service that
> generates a paper on a weekly basis (see [3] for an example), but doing the
> paper layout within TeX was very difficult (I ended up writing some Go code
> that runs pdflatex multiple times and some TeX hacks that measure the
> output of nearly everything before tweaking and composing the final layout
> in a final run). The missing Unicode support in LaTeX still causes
> occasional troubles, I wasn't able to fulfill all wishes from the designer
> (e.g. custom fonts) and it was quite time consuming to define the layout
> within TeX.

Have you used XeTeX, XeLaTex & TexWorks (or TexShop on the
Mac)? XeTeX does Unicode, OpenType, AAT, Graphite etc. I have
been fairly happy with this setup (though I never did anything
very complex in it - just some specs, reports + stuff in
Indian languages). Though I don't use it much now.

> Therefore I sat down yesterday and started writing my own typesetting
> system. An early prototype can be found here [1]. The current
> implementation supports TrueType and OpenType (including Type1) fonts,
> encodes all glyphs directly (to avoid encoding problems and to further
> utilize all features offered by those fonts) and is currently able to
> produce this output [2]. Please note, that you have to provide your own
> fonts, since I haven't found any usable print fonts that are freely
> available yet.

More power to you for IMPlementing but you should ask D.Knuth
about how long his detour to "finish TeX on his sabbatical in
1978" lasted.

[minux]

On Mon, Apr 21, 2014 at 1:41 PM, Bakul Shah <ba...@bitblocks.com> wrote:

More power to you for IMPlementing but you should ask D.Knuth

about how long his detour to "finish TeX on his sabbatical in
1978" lasted.

Things have changed much since that. At that time, he has no choice

but to design his own font, and the whole system that produces his

font design (METAFONT), and PostScript wasn't even invented at that

time. He pretty did everything from scratch.

Now we have pretty standard technology around fonts and document

format that we could leverage, so implementing a new system shouldn't

take that much time.

[Bakul Shah]

Agreed we have good fonts & PDF but just building a typesetting system that can be used for all sorts of things is a pretty nontrivial task. 

[Michael Jones]

Anecdotal: 

I've used SCRIBE from IBM, nroff/troff from BTL, TeX/LaTeX/PDFLaTeX/XeLaTeX all descendants of Donald Knuth's TeX, FrameMaker, and InDesign from Adobe. Each of them works well in their intended uses, and each can be pushed/extended/abused in ways to do almost anything if you're up for the effort.

For my last project (editing/producing a Friend's non-technical book about boating, with many illustrations and simple layout) I used InDesign to produce the cover and XeLaTeX for the body. I could have used InDesign for the whole thing, but chose not to based on minor factors: a few simple equations needed for a display, personal comfort handling structured typesetting as a programming task, and desire to use my existing chapter and index styles. I would never have considered TeX for the cover. It had to be done in Illustrator or InDesign because doing the same subtleties would have been a nightmare in the TeX world. (layers with non-linear fades, color changes, etc.)

The only must for me is that "math must be set with TeX." This is not fair, probably, but I can't even stand to read math set in other tools. I recently read a Cambridge math book sent using troff and it was painful for me. If nothing else, this is proof of Knuth's effect on educated society. TeX is truly as masterwork, as great as TAOCP.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bakul Shah]

On Mon, 21 Apr 2014 12:21:37 PDT Michael Jones <m...@google.com> wrote:
>
> The only must for me is that "math must be set with TeX." This is not fair,
> probably, but I can't even stand to read math set in other tools. I
> recently read a Cambridge math book sent using troff and it was painful for
> me. If nothing else, this is proof of Knuth's effect on educated society.
> TeX is truly as masterwork, as great as TAOCP.

True for me too. Using TeX/LaTeX can often make one pull one's
hair out but the results have been beautiful. TeXWorks/TeXShop
helped quite a bit.

But I do wish there was a better typesetting language with the
power of TeX and one that used its engine underneath. If one
were to combine such a language with Brett Victor's ideas on
direct visualization, you've got something :-)

[adon...@google.com]

On Monday, 21 April 2014 09:20:15 UTC-4, gta wrote:

On Sunday, 20 April 2014 18:19:11 UTC+2, Hotei wrote:

I'm wondering what version of Lout Summerfield used.  I didn't see go in Lout 3.40's prg2lout.c file (a programming language specific code formatter).

 

Lout's source code is interesting.  Looks like a good candidate for conversion to go :-)

Indeed, cannot wait until the c transpiler is ready :)

My understanding is that that tool will be heavily specialized to the job of converting one particular C program to Go, namely the Go compiler, and it will make use of any necessary tricks to that end.  A general C-to-Go translator is not the goal (and is intractable).

[Lubos Pintes]

So does this mean that one day Go will be entirely written in Go?
Dňa 22. 4. 2014 0:41 adon...@google.com
wrote / napísal(a):

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to
> golang-nuts...@googlegroups.com

> <mailto:golang-nuts...@googlegroups.com>.

[ajstarks]

When designing your system please consider typographical best practices like those outlined in [1].

This document also discusses font choices (some freely available). For example, for Deck[2, 3], I use FiraSans[4] from the noted type designer Erik Spiekermann, Other choices include Charter, OpenSans, and ClearSans, and Source Sans Pro. 

Finally, DejaVu Sans, Mono, and Serif [5] is a decent workhorse default set to get you started.

[1] http://practicaltypography.com/

[2] http://github.com/ajstarks/deck

[3] https://speakerdeck.com/ajstarks/deck-a-go-package-for-presentations

[4] http://en.wikipedia.org/wiki/Fira_Sans

[5] http://dejavu-fonts.org/wiki/Main_Page

[Christoph Hack]

Hi Anthony,

Wow, many thanks for all those resources. I really enjoy your first link, especially since I still have to learn a lot of basics in this area. Your font suggestions were also really helpful. FiraSans is one of my favorite fonts now and I can totally understand why you have chosen it for your project. But for mine, a whole font family is probably preferable and since I can't stand the look of DejaVu (I would have preferred a serif font though), I have settled on Source Sans Pro which looks fine in my opinion.

Interestingly, all those free fonts still seem to be far simpler than any other fonts from Adobe. I have modified my otf package to print out some debug information about the supported gsub / gpos features, listed all ligatures and kerning rules, etc. and compared the surprisingly short output to those of commercial fonts. I guess they are worth their money :)

-christoph

[Andy Balholm]

Yes

[Johann Höchtl]

Am Samstag, 19. April 2014 22:41:50 UTC+2 schrieb Christoph Hack:

Hi Gophers,

 

Therefore I sat down yesterday and started writing my own typesetting system. An early prototype can be found here [1]. The current implementation supports TrueType and OpenType (including Type1) fonts, encodes all glyphs directly (to avoid encoding problems and to further utilize all features offered by those fonts) and is currently able to produce this output [2]. Please note, that you have to provide your own fonts, since I haven't found any usable print fonts that are freely available yet.

Many people have already pointed towards lout, which once had an announced successor called Nonpareil, where you can still find documents here:
http://sydney.edu.au/engineering/it/~jeff/nonpareil/

I used lout at varying intensity and found it both powerful and pleasant to write. It comes IMHO out last (after TeX and eqn / troff) when it comes to math though.

Another attempt to escape the TeX bloatware is given by Patoline
http://patoline.org/

[Gour]

Christoph Hack <tux...@gmail.com> writes:

> Ever so often I use pdflatex to write something and get very annoyed while
> doing so. Don't get me wrong, LaTeX is a great and very powerful tool and I
> have used it excessively.

I also like LaTeX's typesetting quality...

> The missing Unicode support in LaTeX still causes occasional troubles,
> I wasn't able to fulfill all wishes from the designer (e.g. custom
> fonts) and it was quite time consuming to define the layout within
> TeX.

...did few books using LyX/LaTeX, but, as you wrote, one would prefer
having more friendly UI to tweak things.

Let me add that Lout sounds interesting, but, afaik, there is no proper
Unicode support which I find 'a must' in 21st century.

Besides LyX/LaTeX and LuaTeX/XeTeX engines, I've also tried ConTeXt
(http://wiki.contextgarden.net/Main_Page), but lack of organized
documentation was a no-go for me.

> Therefore I sat down yesterday and started writing my own typesetting
> system. An early prototype can be found here [1]. The current
> implementation supports TrueType and OpenType (including Type1) fonts,
> encodes all glyphs directly (to avoid encoding problems and to further
> utilize all features offered by those fonts) and is currently able to
> produce this output [2]. Please note, that you have to provide your own
> fonts, since I haven't found any usable print fonts that are freely
> available yet.

Imp looks very intriguing and especially considering it's written in Go
which simply lookes as perfect language for such application.

I just hope you'll find enthusiasm to keep it going as well as that it
will spark some interest within Go community.

All the best!


Sincerely,
Gour

--
Even if you are considered to be the most sinful of all sinners,
when you are situated in the boat of transcendental knowledge
you will be able to cross over the ocean of miseries.